These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is defining not just their APIs, but their schema, and other moving parts of their API operations.07 Sep 2018
This post is part of my ongoing review of the Department of Veteran Affairs (VA) developer portal and API presence, moving on to where I take a closer look at their path to production process, and provide some feedback on how the agency can continue to refine the information they provide to their new developers. Helping map out the on-boarding process for any new developer, ensuring they are fully informed about what it will take to develop an application on top of VA APIs, and move those application(s) from a developer state to a production environment, and actually serving veterans.
Beginning with the VA’s base path to production template on GitHub, then pulling in some elements I found across the other APIs they have published to developer.va.gov, and finishing off with some ideas of my own, I shifted the outline for the path to production to look something like this:
- Background - Keeping the background of the VA API program.
- [API Overview] - Any information relevant to specific API(s).
- Applications - One right now, but eventually several applications, SDK, and samples.
- Documentation - The link, or embedded access to the API documentation, OpenAPI definition, and Postman Collection.
- Authentication - Overview of how to authenticate with VA APIs.
- Development Access - Provide an overview of signing up for development access.
- Developer Review - What is needed to become a developer.
- Individual - Name, email, and phone.
- Business - Name, URL.
- Location - In country, city, and state.
- Application Review - What is needed to have an application(s).
- Terms of Service - In alignment with platform TOS.
- Rate Limits - Aware of the rate limits that are imposed. Production Access - What happens once you have production access. Support & Engagement - Using support, and expected levels of engagement. Service Level Agreement - Platform working to meet an SLA governing engagement. Monthly Review - Providing monthly reviews of access and usage on platform. Annual Audits - Annual audits of access, and usage, with developer and application reviews.
I wanted to keep much of the content that the VA already had up there, but I also wanted to reorganize things a little bit, and make some suggestions for what might be next. Resulting in a path production section that might look a little more like this.
Department of Veteran Affairs (VA) API Path To Production
The Lighthouse program is moving VA towards an Application Programming Interface (API) first driven digital enterprise, which will establish the next generation open management platform for Veterans and accelerate transformation in VA’s core functions, such as Health, Benefits, Burial and Memorials. This platform will be a system for designing, developing, publishing, operating, monitoring, analyzing, iterating and optimizing VA’s API ecosystem. We are in the alpha stage of the project, wherein we provide one API that enables parties external to the VA to submit of VBA forms and supporting documentation via PDF on behalf of Veterans.
[API Specific Overview]
Take a look at the sample code and documentation we have on GitHub at https://github.com/department-of-veterans-affairs/vets-api-clients. We will be developing more starter applications, developing open source SDKs and code samples, while also showcasing the work of other API developers in the near future–check back often.
Spend some time with the documentation for the API(s) you are looking to use. Make sure the VA has the resources you will need to make your application work, before you sign up for a developer account, and submit your application for review.
VA uses token-based authentication. Clients must provide a token with each HTTP request as a header called apiKey. This token is issued by VA. To receive a developer token to access this API in a test environment, please request access.
All new developers must sign up for development access to the VA API platform, providing a minimal amount of information about yourself, business or organization you represent, where you operate in the United States, and the application you will be developing.
You will provide the VA with details about yourself, the business you work for, and your location. Submitting the following information as a GitHub issue, or via email for more privacy:
- Individual Information
- Name - Your name.
- Email - You email address.
- Phone - Your phone number.
- Business Information
- Name - Your business or organizational name.
- Website - Your business or organization website.
- Location Information
- City - The city where you operate.
- State - The state where you operation.
- Country - Acknowledge that you operate within the US.
We will take a look at all your details, then verify you and your business or organization as quickly as possible. Once you hear back from us via email, you will either be declined, or we will send you a Client ID and Secret for access the VA API development environment. When you are ready, you can submit your application for review by the VA team.
You will provide us with details about the application you are developing, helping us understand the type of application you will be providing to veterans. Submitting the following information as a GitHub issue, or via email for more privacy:
- Name - The name of your application.
- Details - Details about your application.
- Terms of Service - Your application is in alignment with ur terms of service.
- Rate Limits - You are aware of the rate limits imposed on your application.
- Working Demo - A working demo of the application will be needed for the review.
- Code Review - A code review will be conducted when application goes to production.
We will take a look at all your details, and contact you about scheduling a code review. Once all questions about your applications are answered, and a code review has been conducted you will be notified letting you know if your application is accepted or not.
Once approved for production access, you will receive an email from the VA API team notifying you of your application’s status. You will receive a new Client ID and Secret for your application for use in production, allowing use the base URL api.vets.gov instead of dev-api.vets.gov, and begin access live data.
Support & Engagement
The VA will be providing support using GitHub issues, and via email. All developers will be required to engage through these channels to be able to actively engage with VA API operations, to be able to maintain access to VA APIs
Service Level Agreement
The VA will be providing a service level agreement for each of the APIs we provide, committing to a certain quality of service, support, and communication around the APIs you will be integrating your applications with.
Each month developers will receive an email from the VA regarding your access and usage. Providing a summary of our engagement. A response is required to maintain an active status as a VA developer, and application. After 90 days of no response, all developer or production application keys will be revoked, until contact is made. Keeping all applications active, with a responsive administrator actively managing things.
Each year developers will receive an email from a VA API audit representative, reviewing your access, usage, and conducting a fresh developer review, as well as review application, access tokens, and end-usage. Ensuring that all applications operating in a production environment continually meet the expected security, privacy, support, and operational standards.
It Is Not Just A Path, But Journey I’m trying to shift the narrative for the path to production into being a more complete picture of the journey. What is expected of the developers, their applications, as well as setting the table of what can be expected of the VA. I don’t expect the VA to bite on all of these suggestions, but I can’t help but put them in there when they are relevant, and I have the opportunity ;-)
Decouple Developer And Application(s) Some of the reasons I separated the developer review from the application review is so that a developer could sign up and immediately get a key to kick the tires and begin developing. When ready, which many will never be, they can submit an application for potential production access. Developers might end up having multiple applications, so if we can decouple them early on, and allow all developers to have a default application in the development environment, but also be able to have multiple applications in a production environment, I feel things will be more scalable.
Not Forgetting The Path After Production I wanted to make sure and put in some sort of process that would help ensure both the VA, and developers are investing in ongoing management of their engagement. Ensuring light monthly reviews of ALL applications using the APIs, and pushing for developers and the VA to stay engaged. While also making sure all developers and applications get annual reviews, preventing what happened with the Cambridge Analytica / Facebook situation, where a malicious application gets access to such a large amount of data, without any regular review. Making sure the VA isn’t forgetting about applications once they are in a production state. Pushing this to be more than just about achieving production status with an application, and also continuing to ensure it is serving end-users, the veterans.
Serving Veterans With Your Application To close, I’d say that calling this a path to production doesn’t do it justice. It should be guide to being able to serve a veteran with your application. Acknowledging that it will take a significant amount of development before your application will be ready, but also that the VA will work to review your developer account, and your application, and ensure that VA APIs, and the applications that depend on them will operate as expected, and always be in service of the veteran. Something that will require a certain level of rigor when it comes to the development, certification, and support of applications across the VA API ecosystem.
I read a lot of API documentation, and help review API portals for clients, and one of the most common rookie mistakes I see made, is people pointing out the obvious, and writing a bunch of fluffy, meaningless content that gets in the way of people actually using an API. When the obvious API industry stuff is combined with the assumed elements of what a company does, you end up with a meaningless set of obstacles that slow API integration down. Here is the most common thing I read when entering an API portal:
“This is an API for querying data from the [Company X] platform, to get access to JSON from our system which allows you to get data from our system into yours using the web. You will need to write code to make calls to our APIs documented here on the page below. Our API uses REST to accept request and provide responses in a JSON format.”
I’ve read API after API that never tells you what the API does. It just assumes you know what the company does, and then goes into verbose explanations of what API, REST, JSON, and other things that should be intuitive if an API is well designed, and immediately accessible via an API. People tend to make to many assumptions about API consumers already knowing what a company does, while also assuming they known absolutely nothing about APIs, and burying actual API documentation behind a bunch of API blah blah blah, instead of just doing and being the API.
It is another side effect of developers, database, and IT folk not being very good at thinking outside of their bubble. It goes beyond techies not having social skills, and is more about them not having to think about other people at all. They just don’t have the ability to put themselves in the shoes of someone landing on the home page of their developer portal, and not knowing anything about the company or the API, and asking themselves, “what does this person need?”. Which I get being something developers don’t think about with internal APIs, but publishing an API publicly, and not stepping back to think about what someone is going to need isn’t acceptable.
Even with my experience, I still struggle to say exactly what needs to be said. There is no perfect introduction to a complex, often abstract set of APIs. However, you can invest a little more time thinking about what others will be needing, maybe run your portal by some external people for a little coherence testing. Most of all, just try to avoid being captain obvious, or captain assumption, and writing content that states the obvious while leaving out most of the critical details you take for granted. It really is the most important lessons we can take away from providing APIs, the ability for them to push us out of our boxes, from behind our firewalls, and have to engage with the real world.
I spend a lot of time talking about the SalesForce API, using it as a reference for where the API evolution began 18 years ago, but it has been a long time since I’ve actually worked with the SalesForce API. Getting up and running with any API, especially iconic APIs that we all should be familiar with, is always an enlightening experience for me. Going from zero to understanding what is going on and actually achieving the API call(s) you want, is really what this game is all about.
As part of some work I’m doing with Streamdata.io I needed to be able to add new leads into SalesForce, and I thought it would be a good time for me to get back into the saddle with the SalesForce REST API–so I volunteered to tackle the integration. The SalesForce API wasn’t as easy to get up and running as many simpler APIs I onboard with is, as the API docs isn’t as modern as I’d expect, and what you need is buried behind multiple clicks. Once you find what you are looking for, and click numerous times, you begin to get a feel for what is going on, and the object model in use becomes a little more accessible.
In addition to finding what you need with the SalesForce REST API, you have to make sure you have a handle on the object structure and nuance of SalesForce itself. For this story, I am just working with one object–Leads. I’m using PHP to work with the API, and to begin I wanted to be able to get leads, to be able to see which leads I currently have in the system:
I will add pagination, and other elements in the future. For now, I just wanted to be able to get the latest leads I have in the system to help with with some checks on what is being added. Now that I can check to see what leads are in the system, I wanted to be able to add a lead, with the following script:
I am only displaying some of the default fields available for this example, and you can add other custom fields based upon which values you wish to add. Once I have added my lead, I wanted to be able to update with a PATCH API call:
Now I am able to add, update, and get any leads I’m working with via the SalesForce API. The project gave me a good refresher for what is possible with the SalesForce API. The API is extremely powerful, and something I want to be up to speed on so that I can intelligently respond to questions I get. I wish the SalesForce API team would spend some time modernizing their API portal and documentation, providing a more coherent separation between the different flavors of their API, and provide OpenAPI driven documentation, as well as Postman Collections. It would have saved me hours of working through their API docs, and playing around with different API calls in Postman before I was able to successfully OAuth, and make my first call against the accounts and leads API endpoints.
While I think SalesForce remains a worthwhile API to showcase when I talk about the history of APIs, and the power of providing web APIs, their overall documentation and approach is beginning to fall behind the times. SalesForce possesses many of the building blocks I recommend other API providers operate, and are very advanced in some of their support and training efforts, but their documentation, which is the biggest pain point for developers, leaves a lot to be desired. I’m used to having to jump through hurdles to get up and running APIs, so the friction for me was probably less than a newer API developer would experience. I could see some of the domain instance url, versioning, and available API paths proving to be a significant hurdle if you didn’t understand what was going on. Something that could be significantly minimized with some simpler, more modern API docs, and OpenAPI and Postman Collections available.
I was on-boarding with the data marketplace Dawex the other day, and I thought their on-boarding process was interesting. It is pretty rigid, requiring users to validate themselves in multiple ways, but it provides some interesting approaches to knowing more about who your API developers are. Dawex has the basic level email and phone number validation, but they have added a 3rd dimension, validating who you are using your LinkedIn profile.
After signing up, I was required to validate my email account–pretty standard stuff. Then I was asked to enter a code sent to my cell phone via SMS–not as common, but increasingly a way that platforms are using to verify you. Then I was asked to OAuth and connect my LinkedIn account to my Dawex profile. I’ve seen sign up and login using LinkedIn, but never using it as a 3rd type of validating who you are, and that you are truly a legitimate business user. I haven’t been verified yet, it says it will take up to 72 hour is what the notification at the top of my dashboard says, but it caught my attention.
Dawes also has a complete, Know Your Customer (KYC) process, which takes the validation to another level, and something I’ll write about separately. I think the social profile validation provides an interesting look at how platforms validate who we are. In an environment where API developers will often sign up for multiple accounts, and do other shady things with anonymous accounts, I can get behind requiring consumers to prove who they are. I also think that providing robust, active, verifiable social media accounts using Github, Twitter, LinkedIn, and Facebook makes a lot of sense. I’d say that all four of these profiles represent who I am as a person, as well as a business.
I have talked about using Github to validate and rate API consumers before. I could see a world where I get validated upon signup using my social profile(s), and the amount of access to an API I am entitled to varies depending on how complete, robust, and verifiable my presence is. I’m not keen on giving up all my data to every API I sign up, but providing access to my profile so they can validate and rank who I am, and then get out of my way when it comes to using their resources, is something I can get behind. I could even envision where you throw Paypal into the mix, and my billing profile is further rounded off, allowing me to consume whatever I desire, and pay for what I used.
I’m working my way through banking and Fintech companies in the UK, and I stumbled across the Starling banking API. I began doing my usual clicking around as I do with any API, looking at the documentation, the getting started, and other primary links. After landing on the Starling getting started page, I have to say that it is the single best example of a getting started page I have ever come across in my time as API Evangelist. It is robust, informative, well laid out, and has everything you need to well, get started.
The Starling getting started page is broken up into six separate sections:
1) Register Your Application 2) Setup Starter Kit 3) Play in the Sandbox 4) Personal Access 5) Going Live 6) Contact Us
Each getting started section has a simple, concise description with relevant visuals and code samples, as well as possession simple action buttons, like sign, login, register application, and the other meaningful things you need to get started. The Starling getting started is going to become my go to example of how to create an API getting started page. You can really tell whoever put it together spent a lot of time refining it, and walking through it until it was 100% complete.
Starling even has a sandbox, marketplace, and a join Slack button. I can’t rave about their approach enough. I’m going to turn it into a case study regarding how to create a getting started page, and showcase on the home page of the site. I wish every API put as much energy into their getting started page as Starling has. It would take the friction out of on-boarding APis, and make it a much more pleasant experience.
I’ve been trash talking APIs that identify me as part of some sort of sales funnel, and automate the decision around whether or not I get access to their API. My beef isn’t with API providers profiling me and making decisions about how much access I get, it is about them limiting profiles making it so I do not get access to their APIs at all. Their narrow definitions of the type of API consumers they are seeking does not include me, even though I have thousands of regular readers of my blog who do fit their profile. In the end, it is their loss, not mine, that they do not let me in, but the topic is still something I feel should be discussed out in the open, hopefully expanding the profile definitions for some API providers who may not have considered the bigger picture.
I’ve highlighted the limiting profiling of API consumers that prevent access to APIs, but now I want to talk about how profiling can be sensibly used to limit access to API resources. Healthy API management always has an entry level tier, but what tiers are available after that often depend on a variety of other data points. One thing I see API providers regularly doing is requiring API consumers to provide more detail about who they are and what they are doing with an API. I don’t have any problem with API providers doing this, making educated and informed decisions regarding who an API consumer is or isn’t. As the API Evangelist I am happy to share more data points about me to get more access. I don’t necessarily want to do this to sign up for your entry level access tier, just so I can kick the tires, but if I’m needing deeper access, I am happy to fill our a fuller profile of myself, and what I am working on.
Stay out of my way when it comes to getting started and test driving your APIs. However, it is perfectly acceptable to require me to disclose more information, require me to reach out an connect with your team, and other things that you feel are necessary before giving me wider access to your APIs, and provide me with looser rate limits. I encourage API providers to push on API consumers before you give away the keys to the farm. Developing tiered levels of access is how you do this. Make me round off the CRM entry for my personal profile, as well as my company. Push me to validate who I am, and that my intentions are truly honest. I encourage you to reach out to each one of your API consumers with an honest “hello” email after I sign up. Don’t require me to jump on the phone, or get pushy with sales. However, making sure I provide you with more information about myself, my project and company in exchange for higher levels of API access is a perfectly acceptable way of doing business with APIs.
Streamdata.io is investing in my API Stack work as we build out their API Gallery of valuable data streaming APIs. I’m powering through hundreds of APIs and using my approach to profiling APIs that I have been developing over the last eight years of operating API Evangelist. I have a large number of APIs to get through, so I don’t have a lot of time to spend on each API. I am quickly profiling and ranking them to quickly identify which one’s are worth my time. While there are many elements that get in the way of me actually being able to obtain an API key and begin using an API, one of the more frustrating elements when API providers require me to jump on the phone with them before I can test drive any APIs.
I’ve encountered numerous APIs that require me talk to a sales person before I can do anything. I know that y’all think this is savvy. This is how business is done these days, but it just isn’t the way you start conversations with API consumers. Sure, there should be support channels available when I need them, but it SHOULD NOT be the way you begin a conversation with us API consumers. I’ve heard all the reasons possible for why companies feel like they need to do this, and I guarantee that all of them are based upon out of date perspectives around what APIs are all about. Often times they are a bi-product of not having a modern API management solution in place, and a team that lacks a wider awareness of the API sector and how API operations works.
In 2018, I shouldn’t have to talk to you on the phone to understand what your API does, and how it fits into what I’m working on. Most of the time I do not even know what I’m working on. I’m just kicking the tires, seeing what is possible, and considering how it fits into my bigger picture. What good does it do for me to jump on the phone if I don’t even know what I’m working on? I can’t tell you much. You can’t share API responses with me. You will able to do less than if you just give me access to APIs, and allow me to make API calls. You don’t have to allow me to make too many calls, just a handful to get going. You don’t even have to give me access to ALL the APIs, just enough of them to wet my appetite and help me understand what it is that you do. This is done using modern API management solutions, and service composition. Giving you the control over exactly how mcuh of your resources I will have access to, until I prove myself worthy.
The APIs I come across that require me to jump on sales call will have to wait until later. I just won’t have the time to evaluate their value, and understand where they fit into my work. Which means they probably won’t ever make it into my project, or any of my storytelling around the work. Which means many of these APIs will not get the free exposure to my readers, helping them understand what is possible. It is just one of many self-inflicted wounds API providers make along the way when they leave their enterprise blinders on, and are too restrictive around their API resources. Sales still has a place in the API game, but the overall API strategy has significantly evolved in the last five years, and is something that is pretty easy to see if you spend time playing with other leading APIs on the market. Demonstrating that these providers probably haven’t done much due diligence about what is out there, which often is just yet another symptom of a poorly run API program, making passing on it probably a good idea.
I came across the new Desktop Underwriter (DU) API from Fannie Mae which provides lenders a comprehensive credit risk assessment data that determines whether a loan meets Fannie Mae’s eligibility requirements. They have a slick new website for the project, with the tag line “building on certainty”, and a smooth HTML story to walk you through what the new DU API can do. While the API seems very exciting, and valuable, the whole production is missing one thing–the API!
I am sure you have to be a partner to get access to the API, but you can tell the whole things is being led by people who have never actually used an API. Otherwise you would give us an API to actually use, and allow us to kick the tires. A hallmark of modern APIs is that you get to play with it. Marketing materials, and a sharp single page application website isn’t enough. We need the documentation, and be able to actually see what the request and response structure is, so that we can better understand the value being generated, and how we will be integrating with it. Without this, there isn’t any value. Of course, you don’t have to make the real API 100% public, you can always create API access tiers, and even deploy a sandboxed or virtualized version of the API and data for new users, protecting your valuable resources–just do not hide the API away from us, and make us consumers beg for access.
When you hide your APIs, you leave first impressions like you did with me. Wouldn’t it be better if my first impression was all about writing a story on how cool your API was, and how all my readers should be using it? Instead, I’m using you as a case of how to not do APIs. There is no reason the Fannie Mae Desktop Underwriter (DU) API can’t be publicly available, allowing us analysts and developers to read the documentation, and kick the tires. Another thing I want to push on back on, is the use of acronyms. I had to Google what DU meant. I know this is ironic, because of API and all, but I work overtime to spell out application programming interface (API) for my (new) users on a regular basis. Please don’t assume all your API consumers will immediately know what DU is. Please help unpack your acronyms, using a glossary, or expanding them inline as you explain what you do. It will make on-boarding much easier for the edges of your target audience.
Anyways, I look forward to some day testing driving the API (someday). I looked for a good five minutes for how to onboard with the API, and eventually gave up. Maybe someone involved in the project will read this, and email me with more information about how I could test drive, so that I can do a proper write up on what Fannie Mae is up to with aPIs. I’m happy to see such large entities working to make their valuable resources available via APIs, but I’m feeling like you might not have done all your homework on what API is, but luckily I’m here as the API Evangelist to help you understand the essence of why APIs work, beyond just the technology and acronym. I look forward to helping you along in your API journey Fannie Mae.
I am a big fan of Jekyll, the static content management system (CMS). All of API Evangelist runs as hundreds of little Jekyll driven Github repositories, in a sort of microservices concert, allowing me to orchestrate my research, data, and the stories I tell across all of my projects. I recommend that API providers launch their API portals using Jekyll, whether you choose to run on Github, or anywhere else using the light-weight portable solution. I have several Jekyll templates I use to to fork and turn into new API portals, providing me with a robust toolbox for making APIs more usable.
My friend and collaborator James Higginbotham(@launchany) has launched a new minimum viable documentation (MVD) template for APIs, providing API provides with everything they need out of the gate when it comes to a presence for their API. The MVD solution provides you with a place for your getting started, workflows, code samples, reference material, with OpenAPI as the heartbeat–providing you with everything you need when it comes to API documentation. It all is an open source package available on Github, allowing any API provider to fork and quickly change the content and look and feel to match your needs. Which in my opinion, is the way ALL API documentation solutions should be. None of us should be re-inventing the wheel when it comes to our API portals, there are too many good examples out their to follow.
I know that Jekyll is intimidating for many folks. I’m currently dealing with this on several fronts, but trust me when I say that Jekyll will become one of the most important tools in your API toolbox. It takes a bit to learn the structure of Jekyll, and get over some of the quirks of learning to program using Liquid, but once you do, it will open up a whole new world for you. It is much more than just a static content management system (CMS). For me, it’s most significant strength has become as a data management system (DMS)??, with OpenAPI as the heart. I use Jekyll (and Github) for managing all my OpenAPI definitions, JSON and YAML files, and increasingly publishing my data sets in this way instead of relying on server-side technology. If you are looking for an new solution when it comes to your API portal, I recommend taking a look at what James is up to.
People often learn through example. Before I’d ever consider myself a software engineer, I’d consider myself a reverse software engineer. 93% of what I know has been extracted from the work of others. Even with 7% being of my own creation, it is always heavily influenced by the work of others. People emulate what they know, what they see, and use. This is why as an API provider you should be showcasing best practices, positive examples, and healthy blueprints of what API consumers could (should) be doing.
You can see this in action with Slack’s best practice blueprints page, where they provide six blueprints of applications that API consumers should be learning from. Slack doesn’t just provide a title, description and image of example applications, it is truly a blueprint–providing diagrams, links to documentation, code samples, and other essential knowledge you will need to successfully develop an application on Slack. Providing six solid examples that anyone can reverse engineer to understand how Slack application development could (should) work.
Slack app blueprints is just one component of a pretty sophisticated getting started section offered as part of the Slack API ecosystem. I am adding application blueprint as a building block to my getting started API research, and adding it as a dimension to my API documentation & SDK research–the overlap in these areas seem like it should be strong to me. Coming across Slack app blueprints, and writing this story has reminded me that I also need to write another piece on the Slack ecosystem, and generate an outline of all the building blocks they are using in their API ecosystem, and create an updated blueprint for successful API operations that other API providers can emulate.
The Azure Marketplace has the ability to test drive anything that is deployed in the Azure Marketplace. As someone who has to sign up for an endless number of new accounts to be able to play with APIs and API services, I’m a big fan of the concept of a test drive–not just for web applications, or backend infrastructure, but specifically for individual APIs and microservices.
From the Azure site: Test Drives are ready to go environments that allow you to experience a product for free without needing an Azure subscription. An additional benefit with a Test Drive is that it is pre-provisioned - you don’t have to download, set up or configure the product and can instead spend your time on evaluating the user experience, key features, and benefits of the product.
I like it. I want more of these. I want to be able to test drive, then deploy any API I want. I don’t want to sign up for an account, enter my credit card details, talk to sales, or signup for 30 day trial–I want to test drive. I want it to have data in it, and be pre-configured for a variety of use cases. Helping me understand what is possible.
I want all the friction between me finding an API (discovery via marketplace) understanding what an API does, test driving, then deployment of the API in any cloud I want. I think we are still a little bit off from this being as frictionless as I envision in my head, but I hope with enough nudging we will get there very soon.
I spend a lot of time trying to figure out what technology does. I read press releases, pitch emails, documentation, and marketing materials trying to get an idea of what is possible. While many of the APIs I come across are intuitive, and just make sense there is still a significant portion of them that leave me scratching my head regarding what it even does.
As developers, it can be easy to think about the SDKs you will need to support API integration with your API, but I think you are making a lot of assumptions about your consumers when you focus your initial energy here. The first step in any API integration begins with IDK and not the SDK. When a potential API consumer comes across your API, the first question to be answered is: what does this API do? If the answer is I Don’t Know (IDK), we have a problem. A problem no amount of SDKs will solve, no matter how many languages you create them in.
Every API begins with an IDK. What does the Twilio API do? What does the Stripe API do? The answer to this initial question for your API cannot be IDK! As soon as I read your press release, land on the home page of your developer area, I should know what your API does in 5 seconds or less. If I walk away from our first interaction with an IDK, the chances I’ll be coming back are pretty slim. Make sure you always address the IDK, before you get to work on your SDK, for your API(s).
I keep an eye on things that are trending daily and weekly on Github because it is a great way to discover new companies and individuals doing interesting things with APIs. While looking at this earlier this week I came across the open guide to Amazon Web Services, a pretty robust, and well organized getting started guide to everything AWS.
Here is their description of this resource out of the leading cloud computing platform:
A lot of information on AWS is already written. Most people learn AWS by reading a blog or a “getting started guide” and referring to the standard AWS references. Nonetheless, trustworthy and practical information and recommendations aren’t easy to come by. AWS’s own documentation is a great but sprawling resource few have time to read fully, and it doesn’t include anything but official facts, so omits experiences of engineers. The information in blogs or Stack Overflow is also not consistently up to date.
This guide is by and for engineers who use AWS. It aims to be a useful, living reference that consolidates links, tips, gotchas, and best practices. It arose from discussion and editing over beers by several engineers who have used AWS extensively.
I find it interesting when API providers invest in authoritative solutions like this, acknowledging the scattered and often out of date nature of blogs, QA sites, forums, and the wealth of other self-service resources available for APIs. Amazon is getting seriously organized with their approach to provider resources for developers--they have been doing this a while, and know where the pain points are.
Amazon's organized approach, the breaking down by service, and the usage of Github are all interesting things I think are worth noting as part of my research. AWS is a tough API pioneer to showcase because they have way more resources than the average API provider, but as one of the early leaders in the API space they possess some serious wisdom and practices that are worth emulating. I'll keep going through their open guide, and see what other patterns I can extract and showcase so that my readers can consider.
As I look through API portals, profiling the building blocks of successful API platform, I'm always looking for bite-size stories for my readers. I was working to complete my Instagram API definition, as part of my photo API research, and their getting started graphic caught my attention.
This three step getting started visual for Instagram is prominently available on the home page, providing a very distilled down version of what I recommend API providers usually accomplish with a dedicated getting started page.
The featured getting started area provides you with the link to register your application, and overview on authentication and authorization, and a link to all the endpoints, so you can start making requests. I am a big fan of there always being a dedicated getting started page for APIs that I am on-boarding with, but I am always a bigger fan when someone distills these building blocks into something that is way easier to understand, and execute--without all the blah blah blah.
I'm going to consider adding a feature getting started building block, in addition to having the dedicated page. When done right, I think they can accomplsh the same thing, but with one less click--reducing any friction we possibly can in the on-boarding process.
Enterprise social API Jive Software wins for the best developer landing page. Normally I tell folks, like I did with Trade.gov last week, that your landing page should have a short, precise description of what you API does—this is some seriously valuable real estate, and you have one chance to make an impression.
I would put Jive Software into the category of more making a first impression, than providing a short description of what the API does:
I wouldn’t recommend this approach for every API provider, but if you can do it right, and make a solid first impression—run with it! As an API Evangelist, and IPA Evangelist, I can seriously get behind a message like “brew up something awesome”. +1
I like how they have apps, docs, APIs, and IPAs as taps, and the taps even work when you mouseover! Seriously, all of this API shit is hard to make entertaining (trust me I know), and when you can provide this human touches while inciting innovation, they can go a long way in breaking the ice with consumers.
Right above the taps Jive Software provides the essential getting started, and register links for their APIs, which signals that beyond the fun facade, they mean business, and want to help you onboard with using their APIs. I haven’t dug deep into the Jive Software API to see what else is going on, but after this first impression it is likely I will be looking around more, now that they made a good first impression.
I’m currently reviewing the Marvel Comics API, and their approach to making their rich content available via a simple web API.
They did a great job deploying the API. To start with they published very simple and honest getting started steps:
- Sign Up - Get an API key
- Link Back - Observe the attribution and linking guidelines when displaying data from the API
- Keep In Touch - Tell us about what you're building and talk to other developers on our community page
- Build Cool Stuff!
I like their style. As a building block, getting started pages are pretty important, yet most API providers provide very dry, technical steps for on-boarding with an API—if at all.
The Marvel Comics API is a refreshing approach to explaining to developers how to get up and running with the API. Its nice to see from such a big brand.
I have a whole list of projects around open data and APIs at the Department of Veterans Affairs (VA). Additionally I have numerous other open data and API projects I'd like to tackle across other federal agencies. As I do with other areas of my work, I needed a standardized way to stabilize the datasets and APis I will need for my projects, in the same way any open data and API provider should do for their consumers.
To help support my work, and hopefully the work of others I wanted to create a baseline portal that I could use at the VA, for showing what is possible when hanging open datasets and APIs, in a full featured portal. The success of any open data and / or API portal starts with the technical building blocks, like data and APIs, but have a set of business and political building blocks that are essential to their adoption and growth.
I've spent the last three years studying the business and politics of APIs. During these three years I've looked at almost 10,000 API developer portals, and I've established a base set of what I consider the building blocks of successful API portals, with a handful in which I consider essential to success. I've always wanted a simple API portal template that would reflect this research, and my new Developer @ VA portal is the first step towards achieving this.
To start with I've included 20 of what I consider essential API building blocks:
- Landing Page
- Getting Started
- Interactive Docs
- Stack Exchange
- Direct Support
- Issue Management
- Branding Guidelines
Beyond this project running using open formats and standards, and being deployable in common cloud environments like Github, Dropbox and Amazon S3, everything is machine readable by default. The portal starts with a sitemap.json, which links to other building blocks that make up the open data and API portal, including a data.json and api.json which provide programmatic access to all resources included in the site:
- data.json - Machine readable JSON file of all data assets using common core metadata for easy access to all data stored in the portal
- api.json - Machine readable JSON, using the Swagger format describing all API resources available within the portal
Next I wanted all essential building blocks of the portal to also be machine readable by default:
- Code - JSON library of all code available for use when integrating with datasets and APIs
- Gallery - A JSON gallery of web, mobile apps and visualizations, widgets, excel and other integrations on data and APis
- Support - A JSON listing of self-service and direct support channels for the portal
- Roadmap - JSON listings of both the future with a roadmap, providing transparency into portal operations
- Changelog - A JSON listing documenting the past with all changes made to the portal
This project is the product of 3 years of research, and 3 days worth of work. So it is rough. I was able to capture my initial vision in 3 days, but still a lot of work to do. I need to let it simmer a little bit, and round off some of the rough edges and apply some polish. Then the next step is to generate a generic template from it, that will become the baseline for any other portal I derive from this work.
If there is anything you'd like to see in the future, go ahead and use any of the support channels available on the site. Once I get the base template set up, I will use that as a top level project to guide the future of this baseline for open data and API portals.
On a regular basis I review my API consumption to evaluate how I’m using various APIs, and what I’m paying for them. I depend on around 20 APIs to make API Evangelist work, and I need to make sure I’m using them to their fullest potential while also being mindful of budget.
As a part of my regular review, I am looking at the differences in pricing between three key services:
- FullContact API - I use FullContact for all my company and individual contact intelligence. I go through phases of light or heavy use depending on research projects I have going on. Full Contact provides me with per API call rates depending on the endpoint and call volume, and they limit me between four packages Trying It out (Free), Getting Started ($19/month), Gaining Traction ($99/month) and Rolling ($499/month)
- Alchemy API - I use Alchemy API to primarily pull text content from blog posts, so I can use it internally for indexing. With Alchemy I get access to three packages Free, Small Business ($250.00) and Basic ($800.00)
- AWS APIs - I use AWS for all my computer, storage, database and DNS API services. With AWS I pay by the resource, bandwidth transfer and storage and the other parts and pieces or specific actions in cloud computing. There are no packages, just modular API resources I use and get billed for.
I’m just one use case, but figured I’d share my thoughts on how I use these three API resources within my little world.
For Full Contact the jump from $19/month to $99/month isn’t too bad. I’ll lump all my processing together into a single month and get lots of work done. So I tend to toggle each month between these two tiers based upon my needs.
For Alchemy API I operate within the free tier and stick with the rate limit of 1,000 API calls per day. If a blog post doesn’t get pulled because I hit my limit, I queue it for another day when I have room within my daily limit. The jump from zero to $250.00 / month is really just too big of a jump for me to make.
My Amazon Web Services bill runs between $250.00 and $1,000.00 / month. Depending on how much traffic, harvesting, processing and other crazy stuff I’m doing. I have a buffet of compute, storage, database, IP address, monitoring,DNS and other cloud computing modules that I have developed and associated with pricing in my head, so that I can make decisions in the moment about whether I can afford to process a bunch of data, launch new API, website etc.
I have a few good friends in the API space that feel API rate limits stimulate creativity, innovation and work-arounds. I agree with that statement, and think every API has to understand its own consumer and make the decision they feel is best. But in the end, I personally like single API resource pricing based upon units, that I can scale infinitely as needed in any moment. I’m am more likely to integrate services into my world when they are independent bite size chunks, with pricing not restricted by other services or service tiers. Each module tends to have different value in my world and I like to make independent decisions about how to use just that resource, disconnected from all the other resources.
The world is starting to look different when I depend on 10-20 APIs vs 1-2 APIs, and I can imagine when I reach the point where I'm depending on 100-200 APIs I will have an even greater need to have API resources be priced independent of other services or limiting pricing tiers.
I’m doing some big data work. Ok it isn’t really, but feels like I should use that label. What I’m actually doing is analyzing the Twitter activity of top APIs.
I have the twitter handles of 100 of the popular web APIs. I have been pulling and analyzing the Tweets from each of these accounts via the Twitter API for some time now.
It is time that I scale this to about 500 API twitter accounts, and also start monitoring mentions of each of these APIs on Twitte, while also performing searches for some keywords associated with each API or a group of APIs.
At this stage of growth, it was clear I was outgrowing the regular Twitter API and since what I’m doing is more big data, than twitter client app, I am looking at the two approved Twitter resellers Gnip and Datasift.
First I went to Gnip, looked at their product and it looked like what I needed. I saw at the bottom of the page, “Get Started”. I recognize that language, clicked on it and filled out their form and I get a “Thank You, Someone Will Be In Touch”.
Ok...so I move on to Datasift. I look at their product and it too, looks like what I need. I see on the page, “Get Started”. I recognize that language, clicked on it, logged on with Twitter, added my email and password, and boom--I’m dropped into a dashboard. I am able to build streams from Twitter using different keywords, Twitter users, etc. I get freemium access, with the ability to put in my credit card and be billed for more.
Next, I got a quick email from a nice fellow at Gnip asking what I was looking to do, and if I had time in the next couple days to talk on the phone. I will definitely talk with them, but by the time I get on the phone with them I will be 2-3 days into developing my solution around Datasift. Unless Gnip has some secret pricing or some magic features Datasift doesn’t have, I probably won’t switch.
That’s the difference between self service and sales oriented web APIs. Which one are you?
One of the essential ingredients of a successful API, is a self-service area to support your API developers. In my opinion this is a no-brainer. You have to have registration, documentation, code samples, forum and other essential API building blocks available to developers in a self-service way--so they can engage with your API 24/7 without asking for access.
I’ve worked hard to make sure the CityGrid API area has a logical navigation, taking developers to the essential information they will need to learn about and integrate with the APIs:
- Getting Started
- Publisher Overview
- Ads by CityGrid Documentation
- Content by CityGrid Documentation
- Code Samples
- Usage Requirements
- Terms and Conditions
- Version History
I keep a regular flow of information going through the blog to make sure there is excellent SEO content, so developers can search for anything at Google, and the blog will support linking to whats relevant in the CityGrid Developer Center.
Even with all of this, I still get some pretty basic questions like:
- I need PHP code samples? - Immediately available on the code samples page.
- What does the API cost? - Says that it is FREE on Getting Started and FAQ.
- Can I save your data in my database? - Says it on the Usage Requirements
- Do I have to show your logo? - Says it on the Usage Requirements
That is just the tip of the iceberg. I’ve posed this question before...are there enough doers in the world to make this whole API vision work? It’s hard to tell if its just my view, while in the pit of despair, or if it is just the way it is.
It seems like there will always be some users who need hand holding, and they refuse to read the self-service material available in the API area. When I’m hacking on an API, I always make sure I search exhaustively before I ask a question, but I’m discovering a new breed of developers who won’t search, won’t ask questions--then when you reach out to them they’ll ask for what they need, no matter how basic.
It seems, that self-service API resources are essential, but there will always be a segment of our audience who aren’t quite doers, they need more hand-holding and information presented to them before they get what they need.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.