Week 12: It has come to this.

Hey everyone. This is my last week for UCOSP updates. I would like to thank all the mentors and those organizing UCOSP for giving me the opportunity to contribute to Freeseer as part of my course work. It has been a great experience and I will continue to keep Freeseer in my sights and hope to continue contributing in the future!

This week:

I learned how to format with restructured text.

I opened two new pull requests.
The first for the API development document. I edited and formatted the API development document I had been working on so that it could become part of the Freeseer development docs. The pull request can be found here.

The second for a developer document on how to get the Freeseer server running, and some more general points on Zeroconf protocol. Found here

Finally I closed my http error standardization pull request.
I still have an open pull request for broadcasting, but it should be complete after code reivew, and I will be sure to close it in the future.

Thanks for everything!

Week 11: Updates

This week I finished the content of my development document. A commentable version is found here. I also made code changes based on code review.

Furthermore, I pushed some preliminary proof of concept code to discover Freeseer hosts on a client's network. There are a few bugs that I tried to get to the bottom of, but I didn't have much luck. Essentially when Freeseer tries to shutdown after running the server discovery process, I will get an exception that I believe is caused by a race condition.

This is the output I get. My guess is that some race condition is occurring when zeroconf.close() fires in the search() method of freeseer/frontend/controller/discover.py. If i add a 'sleep(1)' directly after it, I don't run into this error.

Anyway, the code that actually does the discovering of servers works fine.

Next week,

Obviously things are starting to wind down, I want to continue working towards closing all my pull requests. Furthermore, I have to format and put my API development document in the development wiki. Finally, I have to write a guide on running freeseer server, what dependencies are necessary, etc, which I will do this week.

My only blocks are that my school work load is getting a little insane.

Until next week,

Brenden Doyle

Week 10 Updates

Quick Update:

This week I began putting together a development document for developing APIs and endpoints. The first pass should be up shortly.

I continued working on my open pull requests, in particular the pull request for fleshing out the errors thrown by our REST framework, and I figured it would be a good place to do a bit of refactoring our validation code. Instead of having a tightly coupled validation module with one method per endpoint, I decided to have only a method that takes a request form and a schema to validate it against. In some APIs we will hardcode the schemas into a dict contained by the blueprint (recording api for example), but for request forms that mirror existing classes (Config API for example)  there is a schema method that will generate a schemas at runtime. Either way, the main idea is that we keep the actual schemas away from the validate module, unlike in the past.

I had to make some changes to the test_validate.py module and I ended up making it more pytest-ish instead of unittest-ish.

Next week and beyond:

Finish up development document.

Close all open pull requests

Look at what tests should be added to improve test coverage. Implement new ones if necessary.

UCOSP: Week 9 updates

This week

I continued my research of service discovery options for Freeseer. I made a number of prototypes, got a windows box up and running to make a final decision about what service discovery method to use.

As I elaborated upon last week, there was no perfect broadcasting solution for the Freeseer service. The one that worked the best (avahi) was linux only, the other pure python implementations were very finicky, especially when working as servers. I looked at one final non-Zeroconf no-configuration library called minusconf, that seemed very promising at first, but found that if it was used in any sort of non-standard environment, (multiple NIC cards etc.), it would either work consistently or not at all.

In the end I decided that the best way to accomplish service broadcasting would be to use Avahi on the server side, and a pure python library called python-zeroconf on the client side. This means that only linux boxes will be able to work as headless Freeseer Hosts, but a client can be of any OS. This is a compromise that I am ok with, because Avahi is miles ahead of any pure python zeroconf implementation.

Since we haven't begun developing our Freeseer client yet, there's nothing stopping us from

I put together my implementation of that today, and opened a PR for that issue.

I also took a look at some other work related to revamping the way that HTTP_ERRORS are sent and displayed. I opened a PR for that issue as well. There is now a standard error message associated with any HTTP Error message (404, 401, ...), and an endpoint can decide to supply a description or not. That description will be included in the HTTP_ERROR message JSON returned by the endpoint. It's just to make things a little clearer for developers who will be developing against this REST api, and bring our REST api closer in line with REST best practices.

Next week

I will work on closing these PR's, and will take a look at expanding our test coverage. I also will start thinking of what will go into my API development document, as our time is starting to wind down.

Until next week.

Zeroconf Safari: Week 8 Update

This week: 

This week was quite busy, despite not having committed any code, I had to do a lot of research with regards to broadcasting the Freeseer server across networks. Right now, it is trivial to set up our REST api endpoint so that anyone on our network who knows the IP address of the computer running the server, as well as the port number the server is broadcasting over, they can send api requests from some client. For example, if the Freeseer host machine has an local ip address of 192.168.0.11, and the Freeseer server is running on port 5000, any other computer connected to that network should be able to send api requests to that box with "curl -i -X GET http://192.168.0.11:5000/<endpoint>".

This is only one part of the puzzle however. We want Freeseer servers and clients to be able to be set up and run by people without too much knowledge of ip addresses and networking. What would be ideal is if we could configure things so that Freeseer clients were aware of all instances of Freeseer servers running on a network merely by being connected to it. Even better if we can do this without any configuration necessary. We were aware of at least one way to accomplish this: Zerconf (Zero-configuration networking), implemented by Bonjour on Mac and Windows, and Avahi on Linux. In the most basic terms, Zeroconf is what allows your iTunes to be aware of all iTunes libraries shared on your network auto-magically. So this week I spent researching and prototyping existing implementations of Zeroconf on Python to determine which one was the best to be used by Freeseer.

We had a few critieria for choosing which would be the best:

• Code must work well.
• Must work without any configuration necessary on the part of the user.
• Should refrain from making calls to shell.
• Should be cross platform compatible.
• Should be available through pip.

Basically all we needed was some solution that would broadcast an IP address, a port number, and some other information (name and description of the Freeseer server, for example).

Avahi

Avahi worked really well and painlessly. The only problem is that Avahi is only available for linux, and while headless Freeseer servers are probably only going to be used on linux boxes, we want to run clients on Windows boxes. Therefore, we would need a separate implementation for Windows boxes, and we should be able to accomplish this with only one library.

pybonjour

This library is cross platform, unlike the Avahi solution. I didn't try to make a prototype solution because the code is apparently is very old and unmaintained (last commit 2009), and not particularly well designed according to mtomwing.

mtomwing talked about the possibility of forking and improving.

python-zeroconf

This library actually manages to work fairly well without any external dependencies (relies entirely on socket programming). It caused me a lot of headaches this weekend because Avahi was able to run on a host-only network and python-zeroconf could only work in bridge mode. I have the feeling that even though I managed to get it to work, my gut feeling is that it may only play well with very specific types of network configurations, basically home networks. For example, it makes a call to ip address '4.2.2.1' to get an ip address to bind to. If some network had no DNS server, there is no route to that host and python-zeroconf would falls back to the host IP, (on debian boxes, the host ip is always '127.0.1.1', which useless for us). Avahi has worked extremely well in all the network configurations I've used thus far, including those with no DNS.

I think, the best use for this library would be to use it for clients that don't run avahi. It is just a little too low-level and lacks the robustness of avahi or bonjour to be used for servers.

zeroconf (python module)

This module would work cross platform (ostensibly from looking at the code, I don't have a Windows box to test against). My prototype worked very well. The two biggest issues is that it relies on a number of calls to shell, and it is not installable from pip, meaning we'd have to bring the whole module into our code base.

non-Zeroconf solutions (multicasting, ssdp)

When considering all the problems these three solutions had we tried looking in a different direction and to move away from Zeroconf, and look for some other solution. Without getting into too many details, they all worked well,  but they were all held back by the fact that you needed to specify the interface you wished to broadcast over, which infringes on our first 'must' which is that it must be runnable without dealing with ip addresses and interfaces.

I tried both multicast with socket programming and ssdp (simple service discovery platform) with a small python module, and there was no way that they would be without zero configuration unless we went way beyond the scope of our current project.

Winner? Avahi + python-zeroconf? 

So, essentially all of them have their own positives and heavy drawbacks. I think what will probably happen is we use avahi bindings to create our headless server, and avahi/python-zeroconf for our clients? Perhaps we may just need to fork and refactor pybonjour for our own use.


NEXT WEEK:

Hopefully I will be in the spot where I can put together an implementation this week. I have to discuss where we will go next with the Freeseer mentors. Until next week!

UCOSP Week 7 Updates

This week

This week my pull request for the logic refactor of recording api was finally accepted after a number of code review changes.

Most of my work this week involved researching what I should do next, and how to implement it. I began doing some research on authentication schemes, and have a clear vision for how authentication should be carried out between client and server. After some conversations with Michael Tom-Wing, he noted that it would be better to think about how the server could be broadcasted onto a network and then worry about an authentication scheme later, which will probably rely on shared keys instead of a user/password model. I began looking into what would be the best library for broadcasting,  and I'm still researching. The library that seems the most promising thus far is python-zerconf, which is compatible with both avahi and bonjour, so it should be usable by Windows and OSX as well. I was looking at pybonjour, which is also cross-platform, but does not have support for python 3.

I also thought about a more consistent way to throw and present errors in the REST framework. My plan is to have a table of generic messages associated with each http error in the http error class (400, 404, ... etc.). When an endpoint wants to throw an http error, it then supplies any endpoint specific message it wants to send to the constructor.

I was also sick with the flu Saturday night into Sunday, so I was unable to attend the meeting.

Next week and beyond

I have a clear direction for the last few weeks working on getting our Freeseer REST server to broadcast on a network, so that will be my main focus, as well as small changes to the way errors are thrown by the REST server. As well, I plan on writing a small document about what a developer will need to take note of when developing new endpoints. If I run out of things to do, I always need to expand test coverage. I've modified my proposal to take notes of these changes.

Until next time,
Brenden Doyle

UCOSP Update, Week 6

This week

This week I had a pretty crazy week with midterms. I managed to make changes based on Michael's code review, but I haven't had too much time to think about the code.

Essentially the biggest change this week I made was in the test framework. Our tests require a number of temporary folders to stick files necessary to carry out tests. Before, we would create these temporary folders and delete them on our own, but now the test framework makes use of a pytest fixture called 'tmpdir' that takes care of creating any temporary folders we need to carry out tests and deleting them automatically.

Other than that, there were minor changes here and there to the test framework.

Next week

The main thing I am thinking of right now is changes that can be made to the way we handle errors in the framework, and have a standardized way to handle errors that all new endpoints can adhere to, and how to handle authentication. As soon as my current pull request is closed, I can start to work on either of those.

Until next week.

UCOSP Update, Week 5

Hello everyone.

What I worked on this week.

Implemented shelve in recording.py

This week I began working on an issue where I was trying to figure out the best way to approach tearing down APIs when the rest server stops working. For example, for the recording api, when the server shuts down, the recording api fires some code to write any references to videos to disk. There were a few issues to this:

1. The way to get this code to fire was very hack-ish. It relied on firing code when a kill signal was sent. Furthermore, this code was called from the recording api, and not from some more central location like the server.py class. If we had multiple apis that required some sort of teardown when the server shutdown, it would be better to have a more top down approach. Preferably something that flask provided.

2. There was really no default option provided by Flask to fire teardown code when the rest server shutdown. Since flask provided a number of teardown decorators (post-request teardowns etc.), but provided no teardown decorator that accomplished what I was trying to do, I figured that flask probably actively discouraged this style of teardown.

3. No references to recorded videos were written to disk until after the server shutdown. I was worried that if anything were to go wrong during the shutdown that this teardown code didn't fire, all information in our media dictionary would fail to persist. This was worrisome.

So I decided to make changes to the rest server and recording api so that no teardown code needed to be fired on shutdown of the REST server.

Instead of writing information to disk on shutdown of the REST server, I decided to use a persistent object library called "shelve". Basically you open a "shelve" object, and can treat it as a dictionary (with a few limitations) and when you want your changes to persist, you call sync() on the shelve object.
So, that works well, and no server shutdown teardown code is required for recording api to work properly anymore.

Implemented py.test fixtures in test_server.py

I had to make a number of changes to our REST server's tests so they would work with our changes, and I realized that since I was thinking about the testing framework, that it would be a good time to change our xunit style setup and teardown methods to py.test fixtures. So now test_server.py uses only py.test fixtures to create a baseline for our tests to run against.

Next week

Two things I'm thinking of all the time is the REST server's error framework, and authentication. So depending on when these changes I made this week get merged in, I will work on either of these. I will probably tackle authentication because I already have a lot of work stashed for implementing this. But I am always looking at ways that I could improve the error framework and will be noting things incrementally.

Until next week!

UCOSP Update, Week 4

Hello readers. Another busy week for me with many updates.

What I worked on this week.

My first pull request (link to PR) was merged this week. I learned a lot about how to format your PR commits and summaries properly so it will be accepted. So the next PR should go a lot smoother than this one (apologies to mtomwing :) ).

This week I worked on refactoring the recording api endpoint functions and fixing logic (link to PR). The reason that I didn't do this in the initial framework refactoring PR is that I essentially treated these methods as black boxes and only changed what needed to be changed to decouple the recording api routes from the server.py. Now that that work was complete I could focus on the internal logic of the endpoint functions. I made a number of changes, cleaned up if/else nests, and fixed some faulty logic. These changes are ongoing but should be merged very soon.

Finally, this week I began researching and implementing authentication for the Freeseer server. I read up on REST best practices, and tutorials on how to implement authentication in Flask, and I now have code implementing authentication in the REST server stashed and ready to go into its own branch once my most recent open pull request is merged. In its current state, if one tries to send requests via curl to the API it will give you an authentication error unless you provide -u admin:admin in the command. I need to think more about how we should go about authenticating many users, but it will probably involve tokens generated by the Freeseer server. I'll be focusing on this all week.

Changing priorities

Last week I posted my project proposal, and I've already received some feedback which made me decide to change one of my priorities for the UCOSP project. Worried that if I focused solely on the API framework I would run out of things to do this semester, I considered implementing server-client video streaming, but I was told that there was even more work that could be done to the api framework than what was stated in my proposal. Instead of implementing video streaming, I more than likely will be implementing a way to broadcast our Freeseer REST server over networks, and anything else I discover should be done.

Next week

Next week I will finish implementing authorization for the REST server, and finish the logic refactoring and fixes to the recording api. Looking forward to another busy week.

Until next time.

-Brenden

UCOSP Update, Week 3

Hello everyone, this will be a terse update.

What I worked on this week.

I wrote a project proposal for the work I would like to accomplish this term. The document (WIP) is available here.

I spent most of this week catching up on the assignments and homework I neglected in Toronto, but I was able to sit down Friday and Saturday to research and write a project proposal. Originally I was planning on working on a GUI Client for the REST server, however, my sprint mentor Joseph pointed out that there was a lot of work that needed to be done before we can begin development for the client. After researching RESTful best practices and noting any gaps in the existing REST server, I realized that there was an entire semester worth of work that could be done. So for my project I will implement these best practices. Furthermore, I will research and implement away for the server to stream video to clients. I think that will give me enough work to last me for the semester.

Next week

Next week I begin researching and implementing authorization for the REST server.

Until next time.

-Brenden

UCOSP Update, Week 2

Hello loyal readers.

This week was a fantastic experience. I travelled to Toronto to attend the UCOSP code sprint at Mozilla. I got to meet the other students contributing to the Freeseer project, as well as all students participating in UCOSP. We were also given a tour of the Mozilla building, and given a very inspirational talk about considering the importance of what we create as software developers. All in all it was a fantastic weekend and I am grateful once again for UCOSP covering most of my costs.

Here's what I got accomplished this week:

What I worked on this week:

This week I continued my refactoring of the REST server. Now all methods specific to recording endpoints live in the recording.py module, and methods meant to be shared by all api endpoints live in the server.py module. At this point, the REST server has the same functionality as the it did when developed by bohnje, however, it has been decoupled significantly, and as a result is much more modular. This will make the development of new apis much easier.

Now each API is separated logically and contained in a "Blueprint" which is an object that allows us to extend the REST server app object with functions and endpoints pertaining to that specific API. For example, in the recording api, the functions called to set up and teardown the recording api endpoints is contained in the recording.py Blueprint object, and will fire on the first call to the REST server automatically.

Furthermore, I got a number of comments from Thanh Ha and Michael Tom-Wing about style issues, which resulted in some more changes. I've taken note of their comments and will make more of an effort to follow the PEP8 and Google Python Style guides.

I had a great conversation with Freeseer mentor Joe Yeung about what I should work on for the rest of the UCOSP program. While I was considering developing a GUI Client for the REST server, Joe pointed out that there is still a lot more work that needs to be done on the REST server before we start thinking about clients.

Next week:

For next week, (this week really), I will be writing my project proposal. Essentially, I am going to research best practices for RESTful services, and see what needs to be added to our REST server as a result. Joe pointed out that at the very least, we need to implement some form of authentication, but my research will also point out other things that need to be implemented or improved. 

UCOSP Update, Week 1

Welcome to the first post of my UCOSP updates blog. I will be giving weekly updates about my contributions to the Freeseer project as a participant of UCOSP.

UCOSP (Undergraduate Capstone Open Source Projects) is a program where undergraduate students from universities across North America can contribute to an open source project and receive university credit. I am grateful for being chosen to represent University of New Brunswick at UCOSP, and I'm looking forward to contributing to the Freeseer team this Fall.

First a little bit about me. I am in my last year of my B.C.S. degree at UNB. I tend to be most interested in backend development and scripting languages, which makes Python a good fit for me. One thing I've noticed about studying Computer Science is that the classes are very theoretical and won't necessarily prepare you for software development 'out there'. So, I've been looking for any opportunity to contribute to software development projects, whether it be through a work term placement or an open source project. I'm particularly drawn to open source software development, because I've been an ardent user of open source tools, whether it be Linux, programming languages, DBMS's, or app frameworks, so I recognize that open source software and open source contributors have a positive impact on the world. Therefore, when I first heard about UCOSP, I applied almost immediately. 

So, I suppose what comes next is a status update.

Setting up my development environment:

I spent the first part of my week figuring out how to set up my development environment, after a few false starts trying to get Freeseer working in OSX (some libraries did not play nicely with the otherwise stable Yosemite beta), I decided the best way to go was to install a VM of Xubuntu. The only issue I have is Virtualbox does not want to recognize my MBA's internal camera, so I can't use Freeseer with a camera for now.

Deciding what to project to tackle:

Looking over the issues on freeseer's github, I noticed there was a story to create a GUI freeseer client that could control multiple instances of freeseer servers (github page). I thought tackling this issue would be a good way to contribute, because I have some experience developing REST API servers and clients, and am still interested in learning more about them.  In my most recent work term I worked on a team that was developing a REST API for a SIEM platform. Part of my time was spent developing a functional testing framework (using pytest) for the REST API server, which involved writing an API client in python. With that experience fresh in my mind, I'd like to do something similar for the Freeseer team. I also considered that there may still be some work to do on the server side before development of the clients can take place, and I already have some ideas about what needs to be changed or refactored.

One caveat about the issue I've chosen is that I'm (very) unfamiliar with front-end development in Python. The last time I did any front-end development was in my first year work term, and it was in Visual Studio's WPF (a dark time for me :) ). I do however have a voracious appetite for learning and am up for the challenge.

My progress:

This week I pulled the newly merged code for the REST server, and poked around, seeing how it worked. I noticed that there were a few places where some refactoring could take place. Mainly I noticed that the REST server and the recording api endpoints are very tightly coupled. This would make adding new endpoints unrelated to the recording api somewhat difficult. So I decided for the two weeks leading up to the sprint I would work on refactoring this code. After familiarizing myself with Flask, the backbone of our REST server, I moved the recording api endpoints from the server.py module to its own module called recording.py. I then opened a Pull Request for this work. I probably won't ask for this code to be merged until I'm done refactoring the REST server.

Next week:

Next week I want to move some more of the code found in server.py to the recording.py class. I noticed that all of the setup code for the server, found in the configure() function of server.py, was really code to prepare Freeseer to make use of the recording endpoints and not to setup the REST server. Therefore, I will move that code into recording.py, a more logical place for that code to be kept. My goal is for that recording api configuration code to only be fired if a user calls a recording api endpoint. There are ways to accomplish this using Flask, so it shouldn't be too difficult to do this week. The end result will be a REST api server that is more loosely coupled and modular, and will make the addition of new api endpoints easier.

Future:

After the refactoring is complete (should be done before the Toronto sprint), I will begin my work designing and developing the Freeseer GUI http client.