Improving an RSS Twitter Bot

Shortly after creating the RSS twitter bot to tweet out an unduplicated list of published items, I realized a few issues with the approach: RSS feeds aren’t available for all publications RSS feeds are hyper-specific, which can lead to blind spots if not all feeds are considered RSS feeds are delayed, often by several hours RSS feeds often don’t contain images or video, and don’t contain hashtags and people tags Because of this, the “_once” accounts now retweet the publications’ original tweets instead of composing their own tweets. Read On →

Building an RSS Twitter Bot

Twitter is a great way to keep abreast of breaking news stories. Reputable news outlets, such as the The Washington Post and the The New York Times, maintain Twitter feeds that tweet out their stories as they’re published. Unfortunately, these feeds can contain a lot of noise in the form of duplicates and undesirable content: While some may find value in stories like these, it’s questionable whether they need to see them show up 4 times within 2 days. Read On →

Web Authentication With AWS Lambda and API Gateway

AWS makes building APIs with serverless architecture easy. Using API Gateway and Lambda, you can define functions that interact with databases, make web requests, and process data. While the API Gateway is primarily designed to serve JSON data, you can be configure it to serve plain HTML files and use it as a rudimentary web server. While API Gateway and Lambda have some limitations, such as relatively high latency compared to standalone web servers, and clunky binary data support, for side projects or simple sites it can be a very low-cost and low-maintenance solution. Read On →

Readable, Writable, and Transform Streams in Node.js

Streams in Node.js are very powerful and useful constructs. They are also one of the more difficult concepts in Node to wrap your head around. Over the past few weeks I’ve become more and more familiar with them, and the best way I’ve come to understand them is by creating simple examples of them myself. This post will go through trivial implementations of a readable, writable, and transform stream, and demonstrate how to interact with each and how they interact with each other. Read On →

Setting up Nginx Reverse Proxy for Node.JS on Fedora

One of Nginx’s popular uses is as a reverse proxy for several different Node.js servers. A reverse proxy makes it easy to point to each separate app without having to remember which instance is on what port. Nginx makes this very easy, but Fedora’s SELinux policies make this setup not so straightforward. In this tutorial, I’m going to map 3 Node.js web servers, running on different ports, to different virtual directories on the same domain. Read On →

Understanding Javascript Callbacks

The first step in understanding the concept of Javascript callbacks is to realize that functions are really objects. The thing that makes them special is that you can invoke them (i.e. ‘run’ them). This is done by using ‘.call()’, or simply ‘()’. Because functions are objects, you can pass them as parameters to any other function. Say you have this code: var f = function() { console.log(5); }; f(); This is a pretty simple example. Read On →

Infinite Scrolling in Android

Infinite scrolling has become very popular in recent years. It’s become especially popular on mobile devices for the simple fact that it allows you to fetch new data while accessing data that’s already been fetched. The concept is pretty simple - once the end of the list is detected to be near, a call is made to fetch more data which is then appended to the end of the list. Read On →

Data Join Techniques in JavaScript

When I decided to cut over my festival guide to a standalone site, I needed to figure out a way to do the data operations on the client’s browser that I was doing on the MySQL server. Specifically, I needed to emulate joining tables. The nested loop join is the simplest join you can do in SQL, but it happens to be the most costly. The merge join is a much faster method that can also be easily emulated in JavaScript, but is only advantageous if the sets are sorted (which can offset any performance gain over the nested loop join). Read On →

Text Deduplication in SQL

Data deduplication is essential when importing similar data from different sources. Different providers store data differently, and several variations (both correct and incorrect) exist in the English language for names of people, companies, and entities in general. Deduplication is often made easier if there is a lot of other information associated with the data because it gives you several things to compare to identify a dupe (such as birthday for people, location for company, etc. Read On →

Mapping the Snapchat Data Leak

I’ve been following the Snapchat data leak pretty closely the past few weeks, from the announced weakness to the actual leak of the phone numbers. What I found most interesting about this in particular was that instead of email addresses, password hashes, or credit cards, the leaked data was geographical, mappable data. Digging into what actually leaked, I saw some analysis that suggests that only a fraction of area codes were represented in the leak. Read On →