1. 1
  1. Announcing Jondis

    Working with redis as a database instead of a cache carries additional responsibility. For starters, downtime becomes a serious problem. Up till now, managing this has been somewhat of a pain. Organizations typically deploy a home rolled solution to promote slaves to masters on failure, or handle it manually (since redis is pretty damn stable).

    Recently, redis-sentinel was put into beta, which handles master slave failover of redis instances. This is good for ops, but your application still needs to know when a slave is promoted to a master.

    Enter Jondis. Jondis is a pool for managing your redis master / slave setup that works with redis-py. Given a list of servers, Jondis will learn your topology and if the master server dies, will query the remaining servers to find out which one has been promoted to the master, and reconfigure itself to send requests to the new master instance.

    There’s still a lot to do - currently changing masters without failure isn’t supported, nor is master discovery just by querying slaves, but these features are all on the roadmap and shouldn’t be too much of a problem.

    Check out Jondis - it’s on github and available through pip. We’re currently running Jondis in production on shift.com.

    Enjoy!

  2. Implementing a Python OAuth 2.0 Provider - Part 2 - Authorization Provider

    Last week, I covered the Basics of the OAuth 2.0 Authorization Flow. Today, I will walk through how we used pyoauth2 to set up a minimal Authorization Provider for SHIFT. This post covers setting up endpoints for steps 2 and 5 from the overview.

    The role of the Authorization Provider is to securely generate, validate, and store authorization codes, access tokens, and refresh tokens.

    Routes

    Before getting started, it’s helpful to set up the routes used for OAuth for both a test CustomApp and your application, and have a staging environment to test in. Due to the need for SSL, it is complicated, but not impossible, to test locally.

    The routes for OAuth 2 using pyoauth2 are extremely simple; just forward enough information for pyoauth2 to do it’s job and convert the response to a type appropriate for your application framework. We’ll create an authentication route and a token exchange route, showing code for Flask:

    import flask
    from flask import request
    from app import app
    
    # This class will be defined later in this post
    from shift.oauth.provider import ShiftAuthorizationProvider
    
    # Authorization Code
    # Returns a redirect header on success
    @app.route("/v1/oauth2/auth", methods=["GET"])
    def authorization_code():
    
        # You can cache this instance for efficiency
        provider = ShiftAuthorizationProvider()
    
        # This is the important line
        response = provider.get_authorization_code_from_uri(request.url)
    
        # For maximum compatibility, a standard Response object is provided
        # Response has the following properties:
        # 
        #     response.status_code        int
        #     response.text               response body
        #     response.headers            iterable dict-like object with keys and values
        #
        # This response must be converted to a type that your application
        # framework can use and returned.
        flask_res = flask.make_response(response.text, response.status_code)
        for k, v in response.headers.iteritems():
            flask_res.headers[k] = v
        return flask_res
    
    # Token exchange
    # Returns JSON token information on success
    @app.route("/v1/oauth2/token", methods=["POST"])
    def token():
    
        # You can cache this instance for efficiency
        provider = ShiftAuthorizationProvider()
    
        # Get a dict of POSTed form data
        data = {k: request.form[k] for k in request.form.iterkeys()}
    
        # This is the important line
        response = provider.get_token_from_post_data(data)
    
        # The same Response object is provided, and must be converted
        # to a type that your application framework can use and returned.
        flask_res = flask.make_response(response.text, response.status_code)
        for k, v in response.headers.iteritems():
            flask_res.headers[k] = v
        return flask_res
    

    Ensure that your basic routing is set up properly and returning dummy results before continuing. A good REST client like Postman for Google Chrome will be extremely useful for testing.

    Authorization Provider

    The provider needs to perform many small, independent tasks, mostly validation of important data points, but also storing and retrieving session and token information.

    > Note: All of the major functionality of the authorization provider is implemented in a single class, which I will break out throughout this post. I’ll call this class ShiftAuthorizationProvider. This class acts as a bridge between pyoauth2 and your application logic.

    To start, let’s validate the passed client (CustomApp) information, including the client ID, secret, and redirect_uri. Assume the Application class is a simple ORM. In our case, it’s a MongoEngine model, but should be whatever database you are currently using. All of the validation methods need to return True or False, and a False value will cause the authorization to fail immediately.

    import json
    import re
    from flask import request, session
    from models import Application
    from pyoauth2.provider import AuthorizationProvider
    
    class ShiftAuthorizationProvider(AuthorizationProvider):
    
        def validate_client_id(self, client_id):
            """Check that the client_id represents a valid application.
    
            :param client_id: Client id.
            :type client_id: str
            """
            return Application.find(client_id) is not None
    
        def validate_client_secret(self, client_id, client_secret):
            """Check that the client secret matches the application secret.
    
            :param client_id: Client Id.
            :type client_id: str
            :param client_secret: Client secret.
            :type client_secret: str
            """
            app = Application.find(client_id)
            if app is not None and app.secret == client_secret:
                return True
            return False
    
        def validate_redirect_uri(self, client_id, redirect_uri):
            """Validate that the redirect_uri requested is available for the app.
    
            :param redirect_uri: Redirect URI.
            :type redirect_uri: str
            """
            app = Application.find(client_id)
    
            # When matching against a redirect_uri, it is very important to 
            # ignore the query parameters, or else this step will fail as the 
            # parameters change with every request
            if app is not None and app.redirect_uri == redirect_uri.split('?')[0]:
                return True
            return False
    

    Authorization request validation

    The validate_access method lets pyoauth2 know whether or not a user is logged into your application, and can thus have an access token generated. If someone tries to authenticate to CustomApp without being logged in, the authentication will fail.

    Scope in the context of OAuth represents the categories of access you are granting to the remote app. You can use this to control which third-party applications are whitelisted for different sections of your application. I won’t cover implementing custom scopes in this post, so let’s just return True when no scope is requested.

        def validate_access(self):
            """Validate that an OAuth token can be generated from the
            current session."""
            return session.user is not None
    
        def validate_scope(self, client_id, scope):
            """Validate that the scope requested is available for the app.
    
            :param client_id: Client id.
            :type client_id: str
            :param scope: Requested scope.
            :type scope: str
            """
            return True if scope == "" else False
    

    Token and Code Persistence

    When an authorization code is requested, the request is persisted, and should be valid for a short amount of time. Access tokens are saved for a longer period, and refresh tokens are stored permanently. I’ll assume a StrictRedis client resides in the variable self.redis.

    The data parameter is constructed according to your needs in persist_authorization_code, and only passed through to persist_token_information when the token request succeeds. You should not need to modify data in persist_token_information.

    def persist_authorization_code(self, client_id, code, scope):
            """Store important session information (user_id) along with the
            authorization code to later allow an access token to be created.
    
            :param client_id: Client Id.
            :type client_id: str
            :param code: Authorization code.
            :type code: str
            :param scope: Scope.
            :type scope: str
            """
            key = 'oauth2.authorization_code.%s:%s' % (client_id, code)
    
            # Store any information about the current session that is needed
            # to later authenticate the user.
            data = {'client_id': client_id,
                    'scope': scope,
                    'user_id': session.user.id}
    
            # Authorization codes expire in 1 minute
            self.redis.setex(key, 60, json.dumps(data))
    
        def persist_token_information(self, client_id, scope, access_token,
                                      token_type, expires_in, refresh_token,
                                      data):
            """Save OAuth access and refresh token information.
    
            :param client_id: Client Id.
            :type client_id: str
            :param scope: Scope.
            :type scope: str
            :param access_token: Access token.
            :type access_token: str
            :param token_type: Token type (currently only Bearer)
            :type token_type: str
            :param expires_in: Access token expiration seconds.
            :type expires_in: int
            :param refresh_token: Refresh token.
            :type refresh_token: str
            :param data: Data from authorization code grant.
            :type data: mixed
            """
    
            # Set access token with proper expiration
            access_key = 'oauth2.access_token:%s' % access_token
            self.redis.setex(access_key, expires_in, json.dumps(data))
    
            # Set refresh token with no expiration
            refresh_key = 'oauth2.refresh_token.%s:%s' % (client_id, refresh_token)
            self.redis.set(refresh_key, json.dumps(data))
    
            # Associate tokens to user for easy token revocation per app user
            key = 'oauth2.client_user.%s:%s' % (client_id, data.get('user_id'))
            self.redis.sadd(key, access_key, refresh_key)
    

    Token and Code Loading

    Just as we persisted OAuth information above, we need to implement the loading of the data dictionary from an authorization code (initial OAuth flow) and refresh token (application automatic refresh of access token).

        def from_authorization_code(self, client_id, code, scope):
            """Get session data from authorization code.
    
            :param client_id: Client ID.
            :type client_id: str
            :param code: Authorization code.
            :type code: str
            :param scope: Scope to validate.
            :type scope: str
            :rtype: dict if valid else None
            """
            key = 'oauth2.authorization_code.%s:%s' % (client_id, code)
            data = self.redis.get(key)
            if data is not None:
                data = json.loads(data)
    
                # Validate scope and client_id
                if (scope == '' or scope == data.get('scope')) and \
                    data.get('client_id') == client_id:
                    return data
    
            return None  # The OAuth authorization will fail at this point
    
        def from_refresh_token(self, client_id, refresh_token, scope):
            """Get session data from refresh token.
    
            :param client_id: Client Id.
            :type client_id: str
            :param refresh_token: Refresh token.
            :type refresh_token: str
            :param scope: Scope to validate.
            :type scope: str
            :rtype: dict if valid else None
            """
            key = 'oauth2.refresh_token.%s:%s' % (client_id, refresh_token)
            data = self.redis.get(key)
            if data is not None:
                data = json.loads(data)
    
                # Validate scope and client_id
                if (scope == '' or scope == data.get('scope')) and \
                    data.get('client_id') == client_id:
                    return data
    
            return None  # The OAuth token refresh will fail at this point
    

    Authorization Code and Token Cleanup

    The last essential part of the OAuth flow is discarding unneeded or invalid OAuth data. This keeps your application secure, and eliminates expired or used tokens. If this step is not properly implemented, old access or refresh tokens could be used to compromise your application. These methods will be automatically called by pyoauth2 as appropriate.

        def discard_authorization_code(self, client_id, code):
            """Delete authorization code from the store.
    
            :param client_id: Client Id.
            :type client_id: str
            :param code: Authorization code.
            :type code: str
            """
            key = 'oauth2.authorization_code.%s:%s' % (client_id, code)
            self.redis.delete(key)
    
        def discard_refresh_token(self, client_id, refresh_token):
            """Delete refresh token from the store.
    
            :param client_id: Client Id.
            :type client_id: str
            :param refresh_token: Refresh token.
            :type refresh_token: str
    
            """
            key = 'oauth2.refresh_token.%s:%s' % (client_id, refresh_token)
            self.redis.delete(key)
    
        def discard_client_user_tokens(self, client_id, user_id):
            """Delete access and refresh tokens from the store.
    
            :param client_id: Client Id.
            :type client_id: str
            :param user_id: User Id.
            :type user_id: str
    
            """
            keys = 'oauth2.client_user.%s:%s' % (client_id, user_id)
            pipe = self.redis.pipeline()
            for key in self.redis.smembers(keys):
                pipe.delete(key)
            pipe.execute()
    

    You should now have a working OAuth application! Test the routes manually or by configuring a test application. So far, we have only an Authorization Provider, but no Resource Provider. The Resource Provider will handle a request for a protected resource and respond if the access token checks out.

    The next and final post in this series will cover the remaining steps needed to complete the OAuth protected resource lifecycle.

  3. Implementing a Python OAuth 2.0 Provider - Part 1 - Overview

    Implementing an OAuth 2.0 client is a significant undertaking in itself. Implementing a provider is even more complicated. To make our (and your) lives easier, we’ve open-sourced a Python OAuth 2.0 provider interface, called pyoauth2, which can be easily extended to function as a complete solution for adding OAuth provider functionality to your application. In this post, I will walk through how we set up pyoauth2 for SHIFT.com.

    OAuth 2.0 and pyoauth2 is for you if:

    • You run a Python 2.7-based website or application
    • Third-party apps need to safely connect to it to retrieve your user’s data and interact with your application
    • Third-party apps need to “Login with Your App” without exposing your user’s passwords or other confidential information

    When we have finished this series, we will have created a small example application that will:

    • Prompt for user access to your existing application
    • Receive protected user resources with OAuth validation
    • Stay logged-in to your app permanently and securely through the use of refresh tokens

    Note: only a subset of the very broad OAuth 2.0 Specification is supported by pyoauth2.

    Authorization Overview

    Before writing any code, become familiar with the authorization flow if you have not previously worked with OAuth 2.0. The following diagram is taken from the SHIFT.com authorization provider implementation. Your application’s functionality will be similar, but not necessarily identical to the SHIFT.com example. CustomApp represents any third-party app that would like to connect to SHIFT.com using OAuth. Steps 1-8 are essential and require HTTPS, and any missing portion will likely cause the entire authorization to fail, possibly silently.

    SHIFT.com OAuth 2.0 Implementation

    This diagram was inspired by Facebook’s great explanation of it’s own Login Architecture.

    Note: All requests must be made with HTTPS, and all configuration URLs should begin with https:// as well.

    OAuth Terminology

    Term Meaning
    client_id The third-party application ID pre-registered with the provider
    client_secret The third-party application secret generated by the provider and stored securely by the third-party app and the provider
    redirect_uri The third-party application OAuth 2.0 redirect endpoint

    Step 1: Get Authorization Code

    The OAuth 2.0 authorization process is started with a GET request to the provider server by the user’s browser, usually by clicking a link to the third party app, containing details of the auth request.

    GET https://shift.com/v1/oauth2/auth?
        response_type=code
        &client_id=3faf0fb4c2fe76c1c3bb7d09c21b97c2
        &redirect_uri=https://customapp.com/oauth/redirect
        &shift_team_id=bd6e59b73876f59d13b6fcc944810210
    

    Note: The redirect_uri parameter must be URL encoded, it is shown unencoded here for readability purposes.

    Step 2: Redirect to redirect_uri

    The provider will assess whether or not it can fulfill the request, and redirect to the redirect_uri with either an error, or an authorization code:

    Successful response headers:

    Location: https://customapp.com/oauth/redirect?
        code=hjrZryvgLYo3R833NkHHV8jYmxQhsD8TjKWzOm2f
        &shift_team_id=bd6e59b73876f59d13b6fcc944810210
    

    Step 3: Browser follows redirect

    CustomApp’s server code receives the request and will now perform a series of actions. This browser request resolves via a redirect when step 8 is reached, and the remainder of the steps are performed by CustomApp within this request.

    Step 4: CustomApp server-side request for token

    This request includes the client_secret, which is never sent to the browser. Once this request completes, the OAuth authorization process is complete.

    POST https://shift.com/v1/oauth2/token
    
    Body:
    
        code=hjrZryvgLYo3R833NkHHV8jYmxQhsD8TjKWzOm2f
        &grant_type=authorization_code
        &client_id=3faf0fb4c2fe76c1c3bb7d09c21b97c2
        &client_secret=9643b7c3f59ef531931d39a3e19bcdd7
        &redirect_uri=https://customapp.com/oauth/redirect
    

    Step 5: Token request response

    SHIFT validates the authorization code and other information in the request, and responds with an access and refresh token.

    {"access_token": "2006b436b8f3d83ea291ececc66256c31dcfc67e3",
     "refresh_token": "b669ba34fda054b5b62cd559c9c0676643867e5cd"}
    

    Step 6: Load user information

    This step is not part of the OAuth standard, and is actually the first request made with the OAuth access_token we just received. For SHIFT, CustomApp requests the user using the OAuth Bearer header so it can save the user information in the CustomApp session.

    GET https://shift.com/v1/users/me
    
    Headers:
    
        Authorization: Bearer 2006b436b8f3d83ea291ececc66256c31dcfc67e3
    

    Step 7: User information response

    SHIFT will respond with the information of the user that authenticated through OAuth.

    {"id": "781c5623c31ae512487ab26ea2e01991", "first_name": "Nate", ... }
    

    Step 8: Redirect to App

    At this point CustomApp has obtained from SHIFT:

    • An access_token
    • A refresh_token
    • A user profile

    It is now the responsibility of CustomApp to save all of this information and redirect the user. With pyoauth2, all of the required steps are implemented in 3 classes: AuthorizationProvider, ResourceProvider, and ResourceAuthorization. I’ll explain the purpose and use of each in the remainder of this series.

    Continued in part 2, Implementing the Authorization Provider.

  4. AngularJS: 6 Months Later

    When we started planning the front end for SHIFT, the first question everyone asked was, “what framework should we use?” It was agreed upon early on that we wanted a single-page app built on the same API that we would be providing to third party developers. We wanted a solid front end framework to help us manage a large single-page app. Backbone seemed like the obvious choice because it was lightweight, easy to use, and it had a huge community. However, after doing a few prototyping exercises, we were impressed with the development speed we were achieving with AngularJS. Ultimately we were so impressed that we dove head first into AngularJS, and these are some of the things we found.

    What We Liked

    Two-Way Data Binding

    AngularJS sets itself apart from other frameworks with its innovative approach to data binding. Keeping your data in sync between the JavaScript models and the HTML your users are seeing can be pretty painful, so finding a good way to do this consistently was a big priority for us when choosing a framework. AngularJS provides a few built-in HTML extensions that make instant updates to the HTML when the models change, and it instantly updates the models when the users input new data into the form elements. Forms can be a huge pain, but AngularJS’s two-way data binding actually makes them fun.

    Dependency Injection and Testing

    Using a system called dependency injection, AngularJS provides an easy way to keep track of dependencies. Whenever you create a new AngularJS component, you can have it request each of the services it relies on, and AngularJS will resolve those dependencies at run time. Having AngularJS resolve dependencies for you has an added benefit when you are testing; you can easily replace components, such as the service that does HTTP requests, with a mock service that provides fake data you specify in the test.

    Directives: Extending HTML

    AngularJS’s built-in HTML extensions, or “directives,” provide a lot of power right out of the box. One of the most exciting things about AngularJS is that it gives you access to the same tools the framework’s authors are using to create the built-in directives. With AngularJS directives you can create new elements and attributes that do almost anything you can think of, and create a sort of DSL (Doman Specific Language) out of HTML to describe your applications. For a more in-depth look into how to create your own directives, check out this developer guide.

    Debugging Tools

    When you have an app running and you are working with your favorite browser’s developer console, you may want to inspect a particular part of your application to see what’s going on at runtime. AngularJS provides some great tools for inspecting the DOM elements to see what data is bound to them and where it is coming from. They also have a great Chrome extension that provides even more debugging tools.

    What to Watch Out For

    The Model Layer

    AngularJS doesn’t really provide a solution for managing models. You can use the optional ngResource module, which provides a simple REST interface for grabbing models, but it doesn’t really help you manage models once you get the data. The downside is obviously that you are going to have to figure out how the model layer should work on your own. The upside is flexibility; you can use the model layer of any framework you want and AngularJS won’t care. For SHIFT we built our own data store and model system that has served us pretty well. For our next project we are experimenting with Backbone models and collections, and the results so far have been pretty encouraging.

    Performance

    AngularJS adds more “magic” to the process than other lightweight frameworks such as Backbone. Because of that, it can be easy not to notice how certain patterns may affect the performance of your application. AngularJS makes it very easy, for example, to bind the result of a certain function to the DOM, and have AngularJS update the DOM whenever the result changes. What you might not realize though, is that AngularJS will update this value every time someone interacts with the page. Making sure that your application performs the way you want will be a little less straightforward, because you need to have some understanding of how AngularJS works internally to know how your code will affect performance.

    “Directives are Hard”

    Directives are powerful, but sometimes writing directives is not intuitive. AngularJS documents almost everything you can do with directives on their website, but examples are not always sufficient. We found many times that the only way we could learn how to use a particular feature when creating directives was to read AngularJS’s source code. Improved documentation could really alleviate this issue, and AngularJS has recently released some videos on their blog that do a much better job of explaining directives.

    Documentation

    Documentation is a problem in general with AngularJS. The most commonly used features are very well documented but some concepts like directives, providers, and some lower-level components are not very well documented. This will likely improve as the framework matures. Since we started working with AngularJS, their team has posted a lot of useful videos from their monthly meetups on their blog, explaining such topics as testing and directives in a lot of detail.

    Conclusion

    AngularJS provides a lot of powerful tools for building applications quickly. Within a couple of months we finished an MVP for our product, and had a great foundation on which to keep building. The more we learned about AngularJS the more it helped us out, and it has been a great experience overall.

  5. Advanced Devops with Vagrant and LXC

    Creating and testing new databases that require clustering can be a pain point when trying to do everything on a local machine. Simulating failures or network failures can be difficult or impossible if everything you’re testing is running on the same machine. To better simulate your production environment you can try using LXC (Linux Containers). A linux container is a lot like a Virtual Machine, but shares the host’s Kernel and as a result has very little overhead. A limitation of this is that you can’t mix different environments - for example you can’t run Windows in a container on a Linux host. Theoretically it’s possible to run different Linux distros but so far it seems like there’s a few hiccups doing this.

    Like a full Virtual Machine, a container can be suspended or shutdown easily. It has it’s own IP address and isolated disk space. Unlike a VM you have direct access to the file system from your host, so you can share directories easily by mounting them in the container.

    If you’re looking to set up an environment where this is repeatable and automated, VirtualBox + Vagrant is a winner. Vagrant combines provisioining with Puppet or Chef with VirtualBox to be able to easily launch Virtual Machines tailored to a specific purpose. For this article I’ll assume you’ve already created a VM and are using it.

    Let’s get started. First, get the base LXC packages:

    apt-get install lxc
    

    In order to SSH into your VM, you’ll need a local DNS entry and a network bridge. Fortunately, dnsmasq gets installed when you install lxc through apt, as well as a bridge set up, so you should already have it running. You’ll need to tell your system to send requests to it - just add the IP of the bridge to your /etc/resolvconf/resolv.conf.d/head (you can see the IP of your bridge by looking at ifconfig for lxcbr0)

    nameserver 10.0.3.1
    

    Then reload your resolve.conf

    resolvconf  -u
    

    Lets create a new container creatively called jonhaddad

    lxc-create -t ubuntu -n jonhaddad
    

    Start it up in daemonized mode:

    lxc-start -d -n jonhaddad
    

    And connect:

    ssh ubuntu@jonhaddad
    

    The password is ubuntu.

    You should now be able to connect to your container’s environment. You can install anything you want in here and it’ll be isolated to the container. You can create dozens of containers to simulate dozens of machines, deployments, failures, and behaviors that normally require an entire cluster.

    When you’re finished, you can shut it down like so:

    lxc-stop -n jonhaddad
    

    In the next blog post we’ll be covering customizing your containers for various purposes.

  6. Some of the backend guys turned a conference room into a war room to team code some new graph database tools.

    Larger monitors == increased productivity… so a 55” monitor should == all the productivity, right?

    Brain Hurricane + Tsunami + Volcano = Brain Hurrtsucano

  7. Amazed. Everyday.

    I can’t fully explain how exciting it is working with a group of genius developers building products that improve how people do their day to day work. Not only is the work rewarding and impactful, our team has fun pushing the limits of technologies available (and those that don’t exist, we build). I’ve been able to step back and watch the team collaborate and work together to achieve seriously impressive technological breakthroughs.

    On the backend, we’re working on the bleeding edge of the emerging graph database field, including multiple contributions back to the open source community. 

    On the frontend, we’ve got to be in the top 3 platforms when it comes to codebase size & product complexity based around AngularJS. We’ve also made great progress integrating Backbone.js for its models and AngularJS for the 2-way data binding (blog post coming soon!) which has greatly improved performance and enjoyment of writing code.

    I’ve been doing this for a long, long time and I can say without a doubt that we run one of the most awesome tech stacks in SoCal:

    Front End: AngularJS, Backbone.js, Node,js, CoffeeScript, Stylus, Jade, jQuery, Underscore.js, Jasmine, Testacular, Crossroads.js, and various jQuery plugins

    Back End: Python, Flask, Celery, RabbitMQ, MongoDB, Neo4j, TitanDB, Redis, Cassandra, Elastic Search, Puppet, Fabric, Nose, LXC, Vagrant

    We’ll be posting lots of technically gritty details about what we’re doing, what we’re learning, and anything else we think the community might find helpful. Follow this blog and feel free to reach out to us.

    I’m honored to be a part of this group and I’m truly amazed by what our team does… everyday.

  8. Open Position: Back End Developer

    We’re hiring another python guru to join our team!

  9. Open position: Front End Developer

    We’re actively looking to hire another super smart Front End Developer to join our dev team.

  10. SHIFT Dev Blog

    This is our new home to talk about the Intarwebs