Category Archives: OpenStack

Apache Cassandra, bad assumptions, and unit test shortfalls

This is a story about a problem that took a long time to solve. As in, no one noticed it was a problem for a couple years then it took another year to solve. It involves using the time series database Cassandra, some bad assumptions made by the initial programmers who were more used to relational databases, and a reliance on unit tests that test for a short period of time but miss long term problems. And there is a bonus, somewhat related story at the end.

Continue reading

Kubernetes and Cloud thoughts

I should have a lot more to say on this, but I’m still learning. You see, back in November 2019 my job changed, and I went from being full time developer on an OpenStack based cloud offering and a contributor upstream, to being a full time developer on a Kubernetes based cloud offering and trying to figure out how to fit in to the Kubernetes community. It has been a little slow figuring out how to get in to this new community that follows the directions of Google. I am fortunate to have some coworkers who have some experience in the area, and that has helped greatly. But (and I’m not surprised by this) my first attempt at a simple change upstream to kubernetes/kubernetes has been torpedoed by the oddities of the community. In a nutshell, the published Logging Conventions apparently only apply to some of the code, and that detail is not stated anywhere. So the learning process will continue.

OpenStack Denver PTG – Sept 2018 – Notetaking

I take notes, especially in big meetings, as a way to keep my attention on the meeting (and avoid falling asleep). Sometimes it comes in handy, as my memory isn’t great anyway. Sometimes the benefit is questionable, as my notes might not make sense to anyone else, especially if they aren’t in an OpenStack session.

I took the majority of the notes for the Monasca sessions at the Denver 2018 OpenStack Program Teams Gathering.  Take a look – https://etherpad.openstack.org/p/monasca-ptg-stein – and I did the same last time around – https://etherpad.openstack.org/p/monasca-ptg-rocky .

As a sample, I also took notes for the parts of the OpenStack-Helm sessions that I was interested in.  I didn’t slap them in to the OS-Helm etherpad as I am just peripherally part of the team and didn’t want to break their flow.  But I’m including them below in raw format just for backup/reference.

Continue reading

OpEd: OpenStack wishes for Monasca

This is my first attempt at an editorial post for my blog. I usually just sprinkle in my opinion in whatever I’m writing, but you can see on this site I don’t blog much anyway. But this subject was prompted by a conversation in the Monasca IRC meeting this week (http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-09-05-15.00.log.html) so I thought I’d take a stab at it.

In some ways, this is my wishlist for Monasca and how I would like to see the project get better.  In some ways, it is a bit of a gripe about Telemetry.

Summary: better advertizing of ‘official’ projects, small but active projects can still be useful, and project consolidation.

Continue reading

OpenStack Dublin PTG – Feb 2018 – Thoughts on Wed to Fri

Continuing my thoughts and experience writing about the OpenStack PTG in Dublin, Ireland. Part 1 can be read at OpenStack Dublin PTG – Feb 2018 – Thoughts on First Two Days.

Wednesday through Friday were set aside as focused team work days. As part of the Monasca team (wiki), we really only had enough agenda for two days, and knowing that many of the attendees (including me) had planned to only stay through Thursday. They were two good days of discussion and planning, shortened slightly by the weather (see the next blog post for more on that.

Continue reading

OpenStack Dublin PTG – Feb 2018 – Thoughts on First Two Days

I attended my first PTG (Project Teams Gathering https://www.openstack.org/ptg) this year in Dublin Ireland from February 24 to March 2.  I’ve been working on OpenStack code (and code that uses it or packages and installs it) for a few years now, and recently was made a Monasca Core Reviewer. I attended the OpenStack Summit in Vancouver, which was quite a different experience (should blog about that some time – I have lots of notes I could dig up).

The PTG is different from the Summits.  This was only the third one that has been held, and the first I’d attended.  I had a bit of a sense of what to expect from the meetings after attending some of the developer sessions at the Vancouver Summit, and after the virtual mid-cycle meeting we had for Monasca last year.  But the format was still a bit challenging.

Read below for my take on the first two days, and look for other posts about the rest of the week and weather/travel.

Continue reading

Block heads using Cinder

This is a bit of a case study in the day-to-day work of a software engineer, tied in to a tech tip for OpenStack.  I’m going to throw a few footnotes at the end, so if a term doesn’t make sense bear with the story and I’ll explain it at the end. 😉

In my day job, we are building foundational services for provisioning of servers, networks, and other resources.  We already have a slick platform for this, but are always improving it.  This week as we were working on a recovery story and needed a way to find out if a particular volume had been created, we wanted to query Cinder for some matching metadata.  While we knew it was possible, none of us was sure of the syntax.  So we went to googling for some documentation.  But our searches for “query parameters” to OpenStack Cinder were frustrated.  So we tried looking into the source code for Cinder – its open source and Python, so it is all on github and mostly readable.  That led to some slightly obfuscated code that led us to believe there are filters that can be used.  More googling for “filters” and “search parameters” led to the command line tool for Cinder.

With the proper environment variables in place, the Cinder command line tool works like this:
cinder list –metadata mykey=myvalue1

Getting closer. But we want to use the REST API.  So on to experimentation.  We knew from the limited docs and source code that the volumes URL was what we wanted.

So to try it out, I wrote a quick bash script to invoke curl with the parameters we needed, including the Keystone token. We quickly determined that the JSON metadata query (which can be a partial match to the metadata stored for the volume) needed to be

URL encoded.  Another quick Google and we had an interactive URL encoder up in a browser to run our test JSON string through ten cut and paste into the bash curl script.
But something was still not right. Finally it dawned on me.  When I first wrote the curl command, I wrote it like
http://localhost:8776/v2/{tenantid}/volumes

And naturally when I wanted to query I slapped a / on the end then the query parameters, like

http://localhost:8776/v2/{tenantid}/volumes/?metadata=%7B%22blah%22=%22blah%22%7D

Anyone spot where I went wrong? ‘volumes’ is a resource, not a path, so that slash is sending the request off to nowhere.

Take out the slash.

http://localhost:8776/v2/{tenantid}/volumes?metadata=%7B%22blah%22=%22blah%22%7D

Took longer than it should have, and could have been avoided with a few lines of documentation.

While there is a lot of OpenStack documentation out there, the little technical details can be hard to find.  And this was a case where just getting the right key words into Google wasn’t enough.

OpenStack Cinder is a block storage service used to allocate chunks of hard drive space as virtual volumes for use by virtual machines or attached as external storage to real bare metal servers.
JSON is a markup language like XML but much terser and easier on the whitespace.  I rather like it as long as you have a checker handy to catch missing commas.
Bash is a common shell scripting language.  cURL is a command line toll for sending a request to a URL, which is really handy for repeatedly testing a REST API.  REST APIs are a pattern for remote calls to a server, and I’ll say that the wikipedia article for REST has many more technical words than I want to type out on my 7 inch tablet. 😉

One more reference if you are looking for Cinder documentation.
developer.openstack.org/api-ref-blockstorage-v2.html