Feb 13

Dive Into Python Mirror

Posted on by

I learned to program in Python from DiveIntoPython.org. Unfortunately, the author took down this site and all of his other sites(Dive into Python, Python3, HTML5 and others) in one day and deleted his repositories for these sites. I wrote a script to scrape the Wayback machine (using BeautifulSoup) for these sites, and reuploaded them to the .net versions of these domains. I also scoured the web for the original downloads, and the translations of each site. Currently, my mirrors are the most complete that I can find, though they aren’t finished yet. They are hosted on Amazon S3, so they don’t suck up any server resources and don’t go down. I also host them on GitHub and encourage people to fork them, so they are never lost again. In the repositories, there is a scrape.py that scrapes the sites and an upload.py script that uploads them to S3 through boto.

Note: The sites were written under a permissive license, so I’m not committing copyright infringement by rehosting them.

Feb 13

SQS Mock (Python)

Posted on by

For Djazzee and SiteMon, I initially plan on using Amazon SQS to coordinate between the frontend and the backend. However, when I’m running tests, the delay that often occurs between when something is sent to SQS and it appears to other clients (sometimes upwards of 1 minute for items to show up in the queue), is unacceptable. Therefore, I re-implemented the relevant parts of SQS in Python. It doesn’t have all the functionality of full SQS, but does mostly implement the queues, connection manager, and the adding/removing items from queues. It will evolve as Djazzee/SiteMon evolve, or as people request features. Because of the possible delays in SQS, I may also have to switch to other systems, such as RabbitMQ. Only additional testing will tell.

Feb 13

ChapterSplit

Posted on by

ChapterSplit is a small Python script that interfaces with the Linux utility PDFtk to split up a book into chapters. I have a Calculus book that was too big to load properly on my tablet, so I needed to split it into smaller, chapter PDFs. I wrote a wrapper around PDFtk using Python’s subprocess, and then wrote a loader. The loader reads through a properly formatted text file, which lists the original file, the base for the new file name, and each chapter plus page numbers. It can take separate lists of page numbers. This is really helpful for the Calculus book, because I can add the answers for that chapter to each chapter PDF. I hope to soon extend it to allow multiple PDFs soon, because I have the extended answer book, and I’d like to group it by chapter, instead of having 2 different documents. I also need to clean up some of the error handling and checking on the loaded file. It is kind of rigid right now and not very informative when there is an error.

Jan 04

Implementing Queries and Updates for Minirel (Databases)

Posted on by

This project was fairly challenging, but interesting to complete. I needed to implement the ability to query and update the database (in single user mode only). The main operations were selects, inserts, and deletions. Both the selects and deletions needed to support filtering, to make it more useful. This project really helped me understand how databases work at a bit higher level and taught me a lot queries strategies.

https://github.com/pcsforeducation/Homework/tree/master/cs564/part6

Dec 08

Kernel Filesystem (Operating Systems)

Posted on by

This project was also based on xv6 (the original Unix kernel). It was very challenging to implement, because if anything was wrong, the kernel couldn’t boot. The kernel files would be on the filesystem that I was writing, so the only sensible way was implement both and add a flag to open() with CREAT to specify which filesystem. The original filesystem was pointer based, with indirect pointers. The new filesystem was extent based, like modern filesystems. After that, it wasn’t too hard to implement.

https://github.com/pcsforeducation/Homework/tree/master/cs537/extents

Dec 08

Kernel Pthreads Library (Operating Systems)

Posted on by

Given the xv6 kernel (based on the original Unix kernel), this project required me to implement a shared Pthreads and locking library and write a test suite to test that Pthreads was actually working. It was very challenging to write a kernel level library. Threads are already quite challenging to debug, but they were harder to debug since it was in the kernel. The test suite was pretty straightforward, and just required some long running threaded calls (such as multiple threads locking and incrementing a shared stack), since many bugs don’t crop up in threads until you’ve called it millions of times in a row.

https://github.com/pcsforeducation/Homework/tree/master/cs537/threads/xv6

Dec 08

Kernel System Calls (Operating Systems)

Posted on by

This project was a warm up for the kernel Pthreads and kernel filesystem projects. We were given the xv6 kernel, based off the original Unix kernel (ported to x86), and we needed to implement a new system call. The system call would return the number of times the specified system call (either a different call like open() or the new system call) had been called since boot. It was straight forward, but especially hard to debug, since it was a kernel. One wrong move, and it didn’t boot.

https://github.com/pcsforeducation/Homework/tree/master/cs537/kernel1/xv6

 

Dec 08

Threaded Web Server (Operating Systems)

Posted on by

For this project, our class was given a very basic web server, and needed to add threading. The give code could deal with serving files, HTTP codes, and CGI requests, but that was about it. We needed to add in a worker pool (similar to Apache) and threading to make the workers run efficiently. It was very challenging to debug, like many threaded projects, but it worked in the end.

https://github.com/pcsforeducation/Homework/blob/master/cs537/webserver/server.c

Dec 08

Malloc (Operating Systems)

Posted on by

This project was nicknamed “Pointer Hell”, and made sure I really understood how a pointer works. Basically, I had to implement paged memory with headers in each entry. I also had to implement some basic utilities functions, like a memory dump for debugging. It took a lot of time to get everything just right, but in the end was very helpful to my understanding of main memory and pointers in C/C++.

Dec 08

Simple Shell (Operating Systems)

Posted on by

As another introductory project (project 2) to get us comfortable in C, our Operating Systems class implemented a simple shell. It needed to be able to deal with background processes, piping, calling commands, forking, dealing with errors, and some other basic shell functionality. It was fairly simple, and taught me a lot about checking every system call for more robust programs.

https://github.com/pcsforeducation/Homework/blob/master/cs537/shell/mysh.c