Week 11: Updates

This week I finished the content of my development document. A commentable version is found here. I also made code changes based on code review.

Furthermore, I pushed some preliminary proof of concept code to discover Freeseer hosts on a client's network. There are a few bugs that I tried to get to the bottom of, but I didn't have much luck. Essentially when Freeseer tries to shutdown after running the server discovery process, I will get an exception that I believe is caused by a race condition.

This is the output I get. My guess is that some race condition is occurring when zeroconf.close() fires in the search() method of freeseer/frontend/controller/discover.py. If i add a 'sleep(1)' directly after it, I don't run into this error.

Anyway, the code that actually does the discovering of servers works fine.

Next week,

Obviously things are starting to wind down, I want to continue working towards closing all my pull requests. Furthermore, I have to format and put my API development document in the development wiki. Finally, I have to write a guide on running freeseer server, what dependencies are necessary, etc, which I will do this week.

My only blocks are that my school work load is getting a little insane.

Until next week,

Brenden Doyle

Week 10 Updates

Quick Update:

This week I began putting together a development document for developing APIs and endpoints. The first pass should be up shortly.

I continued working on my open pull requests, in particular the pull request for fleshing out the errors thrown by our REST framework, and I figured it would be a good place to do a bit of refactoring our validation code. Instead of having a tightly coupled validation module with one method per endpoint, I decided to have only a method that takes a request form and a schema to validate it against. In some APIs we will hardcode the schemas into a dict contained by the blueprint (recording api for example), but for request forms that mirror existing classes (Config API for example)  there is a schema method that will generate a schemas at runtime. Either way, the main idea is that we keep the actual schemas away from the validate module, unlike in the past.

I had to make some changes to the test_validate.py module and I ended up making it more pytest-ish instead of unittest-ish.

Next week and beyond:

Finish up development document.

Close all open pull requests

Look at what tests should be added to improve test coverage. Implement new ones if necessary.

UCOSP: Week 9 updates

This week

I continued my research of service discovery options for Freeseer. I made a number of prototypes, got a windows box up and running to make a final decision about what service discovery method to use.

As I elaborated upon last week, there was no perfect broadcasting solution for the Freeseer service. The one that worked the best (avahi) was linux only, the other pure python implementations were very finicky, especially when working as servers. I looked at one final non-Zeroconf no-configuration library called minusconf, that seemed very promising at first, but found that if it was used in any sort of non-standard environment, (multiple NIC cards etc.), it would either work consistently or not at all.

In the end I decided that the best way to accomplish service broadcasting would be to use Avahi on the server side, and a pure python library called python-zeroconf on the client side. This means that only linux boxes will be able to work as headless Freeseer Hosts, but a client can be of any OS. This is a compromise that I am ok with, because Avahi is miles ahead of any pure python zeroconf implementation.

Since we haven't begun developing our Freeseer client yet, there's nothing stopping us from

I put together my implementation of that today, and opened a PR for that issue.

I also took a look at some other work related to revamping the way that HTTP_ERRORS are sent and displayed. I opened a PR for that issue as well. There is now a standard error message associated with any HTTP Error message (404, 401, ...), and an endpoint can decide to supply a description or not. That description will be included in the HTTP_ERROR message JSON returned by the endpoint. It's just to make things a little clearer for developers who will be developing against this REST api, and bring our REST api closer in line with REST best practices.

Next week

I will work on closing these PR's, and will take a look at expanding our test coverage. I also will start thinking of what will go into my API development document, as our time is starting to wind down.

Until next week.

Zeroconf Safari: Week 8 Update

This week: 

This week was quite busy, despite not having committed any code, I had to do a lot of research with regards to broadcasting the Freeseer server across networks. Right now, it is trivial to set up our REST api endpoint so that anyone on our network who knows the IP address of the computer running the server, as well as the port number the server is broadcasting over, they can send api requests from some client. For example, if the Freeseer host machine has an local ip address of 192.168.0.11, and the Freeseer server is running on port 5000, any other computer connected to that network should be able to send api requests to that box with "curl -i -X GET http://192.168.0.11:5000/<endpoint>".

This is only one part of the puzzle however. We want Freeseer servers and clients to be able to be set up and run by people without too much knowledge of ip addresses and networking. What would be ideal is if we could configure things so that Freeseer clients were aware of all instances of Freeseer servers running on a network merely by being connected to it. Even better if we can do this without any configuration necessary. We were aware of at least one way to accomplish this: Zerconf (Zero-configuration networking), implemented by Bonjour on Mac and Windows, and Avahi on Linux. In the most basic terms, Zeroconf is what allows your iTunes to be aware of all iTunes libraries shared on your network auto-magically. So this week I spent researching and prototyping existing implementations of Zeroconf on Python to determine which one was the best to be used by Freeseer.

We had a few critieria for choosing which would be the best:

• Code must work well.
• Must work without any configuration necessary on the part of the user.
• Should refrain from making calls to shell.
• Should be cross platform compatible.
• Should be available through pip.

Basically all we needed was some solution that would broadcast an IP address, a port number, and some other information (name and description of the Freeseer server, for example).

Avahi

Avahi worked really well and painlessly. The only problem is that Avahi is only available for linux, and while headless Freeseer servers are probably only going to be used on linux boxes, we want to run clients on Windows boxes. Therefore, we would need a separate implementation for Windows boxes, and we should be able to accomplish this with only one library.

pybonjour

This library is cross platform, unlike the Avahi solution. I didn't try to make a prototype solution because the code is apparently is very old and unmaintained (last commit 2009), and not particularly well designed according to mtomwing.

mtomwing talked about the possibility of forking and improving.

python-zeroconf

This library actually manages to work fairly well without any external dependencies (relies entirely on socket programming). It caused me a lot of headaches this weekend because Avahi was able to run on a host-only network and python-zeroconf could only work in bridge mode. I have the feeling that even though I managed to get it to work, my gut feeling is that it may only play well with very specific types of network configurations, basically home networks. For example, it makes a call to ip address '4.2.2.1' to get an ip address to bind to. If some network had no DNS server, there is no route to that host and python-zeroconf would falls back to the host IP, (on debian boxes, the host ip is always '127.0.1.1', which useless for us). Avahi has worked extremely well in all the network configurations I've used thus far, including those with no DNS.

I think, the best use for this library would be to use it for clients that don't run avahi. It is just a little too low-level and lacks the robustness of avahi or bonjour to be used for servers.

zeroconf (python module)

This module would work cross platform (ostensibly from looking at the code, I don't have a Windows box to test against). My prototype worked very well. The two biggest issues is that it relies on a number of calls to shell, and it is not installable from pip, meaning we'd have to bring the whole module into our code base.

non-Zeroconf solutions (multicasting, ssdp)

When considering all the problems these three solutions had we tried looking in a different direction and to move away from Zeroconf, and look for some other solution. Without getting into too many details, they all worked well,  but they were all held back by the fact that you needed to specify the interface you wished to broadcast over, which infringes on our first 'must' which is that it must be runnable without dealing with ip addresses and interfaces.

I tried both multicast with socket programming and ssdp (simple service discovery platform) with a small python module, and there was no way that they would be without zero configuration unless we went way beyond the scope of our current project.

Winner? Avahi + python-zeroconf? 

So, essentially all of them have their own positives and heavy drawbacks. I think what will probably happen is we use avahi bindings to create our headless server, and avahi/python-zeroconf for our clients? Perhaps we may just need to fork and refactor pybonjour for our own use.


NEXT WEEK:

Hopefully I will be in the spot where I can put together an implementation this week. I have to discuss where we will go next with the Freeseer mentors. Until next week!