Amazon Elasticsearch Service is a managed service to implement Elasticsearch in AWS. Underlying instances are managed by AWS and interaction with the service is available through API and AWS GUI.
Kibana is also integrated with Amazon Elasticsearch Service. We came across an issue which caused Kibana4 to show the following error message, when searching for *.
Error is not very descriptive.
As Amazon Elasticsearch service is an endpoint only and we do not have direct access to the instances. We also have access to few API tools.
We decided to see what can be found from the chrome browser.
The Chrome Developer Tools (DevTools) contains lots of useful debugging possibilities.
DevTools can be started using several methods.
1. Right click and click Inspect.
2. From Menu -> More Tools -> Developer Tools
3. Press F12
Network tab under DevTools can be used to debug wide variety of issues. It records every requests made when a web page is loading. It captures wide range of information about every request like HTTP access Method, status and time took to complete the request etc.
By clicking on any of the requested resource, we will be able to get more information on the request.
In this case, the interesting bit was under the Preview tab. The Preview tab captures the data chrome got back from the search and store it as objects.
A successful query would look like the image below captured from Kibana3 of public website logstash.openstack.org.
We checked “_msearch?timeout=3000..” and received following errors messages under the nested values (For example “responses” -> “0” -> “_shards” -> “failures” -> “0”)
So the issue is clear, fielddata usage is above the limit.
As per Amazon documentation,
Percentage of JVM heap memory allowed to load a single data field into memory. The default value is 60%. We recommend raising this limit if you are uploading data with large fields.
For more information, see Field data in the Elasticsearch documentation.
Following url documents the supported Amazon Elasticsearch operations.
On checking the current heap usage (second column) of the data nodes, we can see that heap usage is very high,
host ip heap.percent ram.percent load node.role master name
x.x.x.x 10 85 0.00 – m Drax the Destroyer
x.x.x.x 7 85 0.00 – * H.E.R.B.I.E.
x.x.x.x 78 64 1.08 d – Black Cat
x.x.x.x 80 62 1.41 d – Leech
x.x.x.x 7 85 0.00 – m Alex
x.x.x.x 78 63 0.27 d – Saint Anna
x.x.x.x 80 63 0.28 d – Martinex
x.x.x.x 78 63 0.59 d – Scorpio
Following command can be used to increase the indices.breaker.fielddata.limit value. This can be used as a workaround.
Running the command allowed the kibana search to run without issues and show the data.
The real solution would be to increase the number of nodes or reduce the amount of field data that need to be loaded by limiting number of indexes.
Any applications your company runs on premise can also be run in the public cloud. But does that mean they should be?
While the cloud offers well-documented benefits of flexibility, scalability, and cost efficiency, some applications — and especially business-critical enterprise applications — have specific characteristics that can make them tricky to move into a public cloud environment.
That’s not to say you shouldn’t consider the cloud as an option, but you should be aware of the following enterprise application needs before you make any migration decisions:
1. Highly customized infrastructure
Enterprise applications often rely on software components that are uniquely configured: they may need very specific storage layouts and security settings or tight integration with certain third-party tools. That makes it hard to replace them with generic platform-as-a-service (PaaS) alternatives in the cloud.
The same is true on the infrastructure side: application software components often need particular network configurations and controls that aren’t available from a typical infrastructure-as-a-service (IaaS) offering. (An example would be the way Oracle Real Application Clusters have to allow the cluster software to manipulate network settings, such as controlling IP addresses and network interfaces.)
2. Tightly coupled components
Today’s cloud application architectures are based on “microservices” — collections of services that perform specific tasks. When combined, these answer the whole of the application requirements. With enterprise applications, there are so many interdependencies between the various software components that it can be extremely difficult to change, upgrade, move, or scale an individual component without having a huge impact on the rest of the system.
3. Siloed IT departments
Enterprise applications are usually supported by siloed enterprise IT operations — DBAs, system administrators, storage administrators, network administrators and the like — each with their own responsibilities. Cloud deployment, on the other hand, requires much greater focus on collaboration across the IT environment. This means breaking down traditional silos to create full-stack teams with vertical application ownership. Some teams are likely to resist this change as they could end up with significantly less work and responsibility once the management of application components has shifted to the cloud vendor. So migrating to the cloud isn’t just a technical decision; it has people-process implications, too.
4. Costly infrastructure upgrades
Every company knows upgrading enterprise applications is a major undertaking and can often cause downtime and outages. This is true when the application stays inside your own data center — and doubly so when it moves to a cloud provider due to how long it takes to move massive amounts of data through the Internet and risks associated with unknown issues on the new virtual platform. For these reasons, significant financial commitment is often required to build and maintain an IT team with the right skills to do upgrades quickly and effectively as well as maintain the system.
5. Inflexible licensing models
The components used in enterprise applications are often proprietary products with licensing models that are not compatible with the elasticity of the cloud. For example, many Oracle licenses are for legacy applications and can used only on particular systems. Transferring those licenses to a cloud-based infrastructure is not an easy task.
In addition, perpetual software licenses are often not portable to the typical pay-as-you-go model used by most cloud providers. Plus, most software vendors don’t have any incentive to transition their customers from locked-in perpetual licenses with a steady maintenance revenue stream to a model that allows them to switch to a competitive product at any time.
Even though the nature of enterprise applications makes them difficult to migrate to the cloud, the benefits of doing so — in costs savings, availability, and business agility — still make it a very compelling proposition. In my next blog, I’ll take a look at some of the paths available to you should you decide to move your enterprise applications to the public cloud.
For more on this topic, check out our white paper on Choosing the Right Public Cloud Platform For Your Enterprise Applications Built on Oracle Database.
True inspiration comes in many forms and I consider myself fortunate to be inspired every day by the women around me. International Women’s Day is the perfect opportunity to reflect on the women in our lives who positively influence us.
This post is my heartfelt thank you to the women in my ‘circle’ who have made an indelible mark in my life. They are women who continue to inspire, challenge and motivate me.
The women on Pythian HR’s team: These women continually teach me valuable life lessons. They are mothers, partners, sisters, care providers, aunts, cousins, and friends to many. They are strong, spirited, supportive and have generous natures that are contagious. They demonstrate an unwavering commitment to working hard, they’re incredibly talented and they have a steady focus on doing what’s best for our employees. These women go above and beyond and approach every puzzle with optimism.
My mother: My mother is the most positive and ‘glass half full’ person that I know. She is a person who never fails to find the bright side to life’s most thought-provoking issues and one of her favourite questions to ask her loved ones is “Are You Happy?” (Spoiler alert: she’s not satisfied unless the answer is a truthful “yes”). Her love, guidance and support have helped sustain me through so much and over the years she has evolved into my BFF.
My friend, Jen: Jen is a breast cancer survivor who decided to fight back and co-found Vixens Victorious. In October 2015, the dynamic duo of Vixens Victorious successfully launched Lights! Camera! CURE! which showcases female film makers from Canada and the proceeds go to support the Ottawa Regional Cancer Society. Jen’s positive spirit and take charge attitude empowers everyone who meets her.
My friend, Kate: Kate moved to Canada with her three month old daughter to start a new journey with her husband. She took the initiative to make new friends, develop a network and often navigate a new city on her own when her partner travelled for work. Kate isn’t one to complain about adversities in life; she is courageous and gratefully embraces her adventure.
My fitness trainer Jules: Jules gets out of bed every morning, puts on her workout gear and travels across Ottawa to provide the most fun and effective workouts to her clients. She generously shares her own personal health journey and always finds a way to connect with her clients so they can experience the one on one attention they need. She is full of greatness.
Our family physician, Dr. Judy: Dr. Judy’s medical practice is thriving because of her commitment to patient care. She ensures you are her priority in the moments that you are with her. She makes each of her patients feel important, cared for and heard. Dr. Judy emulates a kind and caring nature that everyone could benefit from.
My neighbor, Anne Marie: In her late forties, Anne Marie taught herself to swim so she could begin competing in triathlons. She now travels internationally to compete in races. I’m inspired by her hard work, determination and strategic ability to set and meet goals.
The influences (sometimes subtle) of these women make an impact on how I choose to live my life. I am thankful for all of them.
On this International Women’s Day, I encourage you to think about who inspires you and why. Bonus points if you honour them with words of appreciation!
Years ago, I got burned by an “April Fools” joke published by Steve Jones on sqlservercentral.com. He republished it as one of his favorites here.
Naturally, I had to rub my eyes today when I read that Microsoft announced that SQL Server 2016 would be coming to Linux.
There were mixed reactions on the internal SQL Server teams. I was afraid to respond to the thread, fearing I would get burned again. I quickly checked the date to confirm that the article hadn’t been resurrected.
One of the sentiments expressed in our internal chatter was that some of the DBAs love “Satya’s Microsoft” and I agree. I like what they’re doing, but I am very skeptical about the move to port SQL Server onto Linux.
I doubt this will enable new markets and new customer bases. I don’t think there are any large organizations who will suddenly decide to adopt the product because it will run on Linux.
One theory was that this move was to attract new developers who want to deploy multi-platform tech stacks. That could certainly be right, but I think PaaS satisfies that and many of the startup natures.
Other articles I read theorized it was a move towards SQL Server on Linux-powered containers.
I’m wondering what this will mean for future features. Will PowerShell or .NET be ported to Linux? What will change in the security model? Will clustering be available? Will a more RAC-like feature be available?
These are very interesting times and while this wasn’t a move that I was pining for, or even expected, I am excited to see where this is going.
I “applied” to test one of the early versions, and you can too.
What do you think? Are you excited about running SQL Server on Linux? When would you choose Linux over Windows?
The Percona Live Data Performance Conference in Santa Clara is being held April 18-22, 2016. It is quickly approaching, and Pythian is going to show you how we Love Your Data in a big way!
We have an awesome lineup of speakers this year:
- Alkin Tezuysal, Okan Buyukyilmaz, and Emanuel Calvo will be presenting the Break/Fix Lab tutorial. This is becoming a standard so if you haven’t had the opportunity to participate, don’t miss it!
- Christos Soulios will be presenting a tutorial on MongoDB design patterns with Pythian alum Nik Vyzas and Percona’s Roman Vynar.
- Derek Downey is co-presenting with HashiCorp’s CTO Arman Dadger on using vault to decouple secrets from your applications.
- Martin Arrieta will be hosting a Birds-of-a-Feather session on the best practices of running XtraDB Cluster with HAProxy.
- John Schulz will show you how to shard effectively whether you run MySQL, MongoDB or Cassandra.
Mark these down in your schedule because you are not going to want to miss any of them! Although, you might have a tough time choosing between the tutorials if you can’t clone yourself.
RSAConference 2016 Where the world talks security
40,000 attendees, 500+ vendors and 700 sessions
RSAC is my annual check in to learn new approaches to information security, discover new technology, learn from industry experts and build my community.
In the three years that I have been attending RSAC, I have learned that Pythian is unique and so are our clients. Each year, we continue to improve our security program with our clients in mind.
RSAC Day 1
It’s Day 1 of the RSAConference 2016. Monday’s are typically a quiet day with vendors setting up in the expo halls, conference staff getting organized, attendees registering and a few press/analysts looking for optimal interview spots. It has been the calm before the storm of attendees descend on San Francisco and RSAC.
This Monday was a whirlwind of activity; CSA Summit, DevOps Connect, Information Security Leadership Development and IAPP: Privacy and Security to name only a few. Chances are you may have missed sessions if you weren’t early enough.
Privacy and Security were hot topics given the European General Data Protection Regulation (GDPR) agreement reached December 2015.
Once again, Pythian is organizing an event that by now may be considered a tradition: The MySQL community dinner at Pedro’s! This dinner is open to all MySQL community members since many of you will be in town for Percona Live that week. Here are the details:
What: The MySQL Community Dinner
When: Tuesday April 19, 2016 – 7:00 PM at Pedro’s (You are welcome to show up later, too!)
Where: Pedro’s Restaurant and Cantina – 3935 Freedom Circle, Santa Clara, CA 95054
Cost: Tickets are $40 USD, Includes Mexican buffet, non-alcoholic drinks, taxes, and gratuities (see menu)
How: Purchase your ticket below or RSVP through Eventbrite
Maintaining mission critical databases on our pitchfork wielding brother, the “Daemon” of FreeBSD, seems quite daunting, or even absurd, from the perspective of a die-hard Linux expert, or from someone who has not touched it in a long time. The question we ask when we see FreeBSD these days is “why?”. Most of my own experience with FreeBSD was obtained 10-15 years ago. Back then, in the view of the team I was working on, a custom compiled-from-source operating system like FreeBSD 5.x or 6.x was superior to a Linux binary release.
Package managers like YUM and APT were not as good. They did not always perform MD5 checks and use SSL like today’s versions. RedHat wasn’t releasing security updates 5 minutes after a vulnerability was discovered. Ubuntu didn’t exist. Debian stable would get so very old before receiving a new version upgrade. FreeBSD was a great choice for a maintainable, secure, free open source UNIX-like OS with tight source control and frequent updates.
Most people do not understand why FreeBSD remains a great choice for security and stability. The main reason is that the entire source of the base OS and the kernel (not just the kernel) are tightly maintained and tested as a whole, monolithic, distribution.
FreeBSD 10.2 is different than versions I worked on many years ago, in a good way, at least from the standpoint of getting started. First, “pkg” has gotten quite an overhaul, making installing packages on FreeBSD as easy as with YUM or APT. portsnap and portmaster make port upgrades much easier than they used to be. freebsd-update can take care of wholesale updates of the operating system from trusted binary sources without having to “build the world”. These are welcome changes; ones that make it easier to get to production with FreeBSD, and certainly made the task of rapidly building and updating a couple of “lab” virtual machines easier.
In my effort to get re-acquainted with FreeBSD, I hit some snags. However, once I was finished with this exercise, FreeBSD had re-established itself in my mind as a decent flavor to host a mission critical database on. Open Source enthusiasts should consider embracing it without (much) hesitation. Is there some unfamiliar territory for those who only use MySQL on MacOS and Linux? Sure. But it is important to remember that BSD is one of the oldest UNIX like operating systems. The OSS world owes much heritage to it. It is quite stable and boring, perhaps even comfortable in its own way.
Problem 1: forcing older versions of MySQL
I needed to install MySQL 5.5 first, in order to test a mysql upgrade on FreeBSD. However, when installing percona-toolkit either via “pkg install” (binary) or /usr/ports (source), the later 5.6 version of the mysql client would inevitably be installed as a dependency. After that point, anything relating to MySQL 5.5 would conflict with the 5.6 client. If I installed in the opposite order, server first, percona-toolkit second, the percona-toolkit installation would ask me if it is OK to go ahead and upgrade both server and client to 5.6.
TIP: don’t forget make.conf
Once I added MYSQL_DEFAULT into make.conf, the installations for MySQL 5.5 became seamless. Note: if you want another flavor of MySQL server such as Percona Server, install the server “pkg install percona55-server” prior to “pkg install percona-toolkit” so that the client dependencies are met prior to installation.
Problem 2: Some tools don’t work
pt-diskstats does not work, because it reads from /proc/diskstats, which does not exist on FreeBSD. Other favorites like htop don’t work right out of the box. So far I have had good luck with the rest of the Percona toolkit besides pt-diskstats, but here’s how you get around the htop issue (and perhaps others).
TIP: Get the linux /proc mounted
# kldload linux
# mkdir -p /compat/linux/proc
# mount -t linprocfs linproc /compat/linux/proc
to make permanent:
# vi /boot/loader.conf (and add the following line)
# vi /etc/fstab (and add the following line)
linproc /compat/linux/proc linprocfs rw 0 0
As you may have determined, these commands ensure that the linux compatibility kernel module is loaded into the kernel, and that the linux style /proc is mounted in a different location than you might be used to “/compat/linux/proc”. The FreeBSD /proc may also be mounted.
Problem 3: I want bash
# pkg install bash
… and once that’s done
# pw user mod root -s /usr/local/bin/bash
…and repeat ^^ for each user you would like to switch. It even comes with a prompt that looks like CentOS/RHEL.
Problem 4: I can’t find stuff
BSD init is much simpler than SysV and upstart init frameworks so your typical places to look for start files are /etc/rc.d and /usr/local/etc/rc.d. To make things start on boot, it’s inevitably a line in /etc/rc.conf.
In our case, for MySQL, our start file is /usr/local/etc/rc.d/mysql-server. To have MySQL start on boot, your rc.conf line is:
If you do not wish to make MySQL start on boot, you may simply say
Notes on binary replacement
Please note, just like in the Linux world, MariaDB and Percona Server are drop in replacements for MySQL so, the startfiles and enable syntax does not change. Your default location for my.cnf is /etc/my.cnf just like in the rest of the known universe.
This command lists all installed packages.
pkg info -a
pkg remove and
pkg install to add new versions of your favorite mysql software.
I ran into no greater issues with pkg than I would with yum or apt doing binary removals and installations, and no issues at all with mysql_upgrade. Remember: If you had to alter make.conf like I did earlier, remember to update it to reflect versions you want to install.
For those who like ZFS, the FreeBSD handbook has a very detailed chapter on this topic. I for one like plain old UFS. It might be the oldest filesytem that supports snapshots and can be implemented very simplistically for those who like low overhead.
Happy tinkering with FreeBSD and MySQL, and thanks for reading!
On a scorching November day in 2012, Sydneysiders were bracing themselves for yet another heat wave when all of a sudden they became pleasantly surprised as an elastic cloud occupied the tech skies. On November 12, 2012, Amazon announced the New Asia Pacific (Sydney) Region in Australia.
Before that, Australian customers had to reach out to Japan or Singapore for their cloud needs. That was not really feasible, as it increased up-front expenses, long-term commitments, and scaling challenges. Amazon recognized that and Sydney became another region in the world.
They have now taken it a step further. They have rendered a new Availability Zone (AZ) in Sydney. Availability zone (AZ) is basically an isolated location within data centre regions from which public cloud services originate and operate.
The new availability zone is ap-southeast-2c. This is all set to provide enhanced performance and sociability to Australian customers. This will enable them to fully leverage the potential of technologies like Lambda, the Elastic File System shared filesystem, and Amazon RDS for MS SQL Server.
Pythian’s established presence in Australia and New Zealand coupled with round the clock and world class support for AWS, SQL Server, and other cloud technologies, enables it to support Australian and New Zealand customers from the word go.
Recently we’ve received an alert from one of our clients that running threads are high on one of their servers. Once we logged in, we noticed that all the selects were waiting for table level read lock. We scrolled through the process list, and found the selects which were causing the problems. After killing it, everything went back to normal.
At first we couldn’t understand why the query took so long, as it looked like all the others. Then we noticed, that one of the WHERE clauses was strange. There, we found a SLEEP(3) attached with OR to the query. Obviously, this server was the victim of a SQL injection attack.
I think most of us know what SQL injection is, but as a refresher, SQL injection is when someone provides malicious input into WHERE, to run their own statements as well.
Typically this occurs when you ask a user for input, like username, but instead of a real name they give you a MySQL statement that will be run by your server without you knowing it.
Based on the picture, let’s see a few examples.
We have a simple table:
mysql> describe post; +-------+------------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------+------------------+------+-----+---------+----------------+ | id | int(10) unsigned | NO | PRI | NULL | auto_increment | | test | varchar(127) | YES | | NULL | | +-------+------------------+------+-----+---------+----------------+ 2 rows in set (0.00 sec) mysql> select * from post; +----+--------+ | id | test | +----+--------+ | 1 | text1 | | 2 | text2 | | 3 | text3 | | 4 | text4 | | 5 | text5 | | 6 | text6 | | 7 | text7 | | 8 | text8 | | 9 | text9 | | 10 | text10 | +----+--------+ 10 rows in set (0.00 sec)
Lets run a select with LIKE, which we know for sure won’t have a match:
mysql> select * from post where test like '%nomatch%'; Empty set (0.00 sec) But what, happens if we don’t filter the inputs and someone wants to get all the data? mysql> select * from post where test like '%nomatch ' || '1==1' && '1%'; +----+--------+ | id | test | +----+--------+ | 1 | text1 | | 2 | text2 | | 3 | text3 | | 4 | text4 | | 5 | text5 | | 6 | text6 | | 7 | text7 | | 8 | text8 | | 9 | text9 | | 10 | text10 | +----+--------+ 10 rows in set, 2 warnings (0.00 sec)
That was a very mild injection, but it could be much more malicious: we could drop another table!
mysql> show tables; +----------------------+ | Tables_in_injecttest | +----------------------+ | game | | post | +----------------------+ 2 rows in set (0.01 sec) mysql> select * from post where test like '%nomatch'; drop table game;-- %'; Empty set (0.00 sec) Query OK, 0 rows affected (0.28 sec) mysql> show tables; +-----------------------+ | Tables_in_inject_test | +-----------------------+ | post | +-----------------------+ 1 row in set (0.00 sec) mysql>
If we don’t know the name of the table, we can still cause trouble by blocking access to the database
If we insert SLEEP() in the WHERE part, then it will be executed for every matching row… if we inject it like: “OR SLEEP(n)”, it will be executed to every row in the table!
Okay, this will be “just” a long running select. It shouldn’t cause much trouble thanks to InnoDB and transaction isolation, unless something needs a table lock.
Some common examples of what causes table locks are:
- explicit lock table
- insert/update/delete on MyISAM
- ALTER table on InnoDB
Once statements start waiting for lock on the table, all proceeding selects will wait for the previous locking statement to finish
Terminal 1: mysql> select * from post where test like '%nomatch ' OR sleep(300) AND '1%'; …. Terminal 2: mysql> alter table post engine=innodb; ... Terminal 3: mysql> select SQL_NO_CACHE count(*) from post; … Processlist: +----------+----------------------+-----------+--------------------+---------+-------+---------------------------------+-----------------------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +----------+----------------------+-----------+--------------------+---------+-------+---------------------------------+-----------------------------------------------------------------------+ | 17170817 | root | localhost | janitest | Query | 19 | User sleep | select * from post where test like '%nomatch ' OR sleep(300) AND '1%' | | 17170918 | root | localhost | janitest | Query | 11 | Waiting for table metadata lock | alter table post engine=innodb | | 17170957 | root | localhost | janitest | Query | 4 | Waiting for table metadata lock | select * from post | +----------+----------------------+-----------+--------------------+---------+-------+---------------------------------+-----------------------------------------------------------------------+ 3 rows in set (0.00 sec)
As we see in the example, ALTER table will wait until it can get a lock on post table, and this blocks every other select from now on to the table.
Or, if you are using MyISAM table, a simple update/insert will block access to the table, because it needs table level lock during them.
There are several ways to secure yourself from SQL injection.
- First of all, validate the input. If you expect only letters and numbers, filter it with regexp for example, to make sure there are no special characters there. Also escape the inputs on application side; programming languages have built-in function to do that (eg.: mysql_real_escape_string() in PHP)
- Use prepared statement! It won’t allow 2 clause if you specified only 1. When you use prepared statements, the variables are transmitted as MySQL variables. Even if the string is not escaped, it will end up in one variable, and MySQL treats is as a longer string.
(For more details see: http://dev.mysql.com/doc/refman/5.7/en/sql-syntax-prepared-statements.html )
- Use a tool like MySQL Enterprise Firewall, which is a plugin for MySQL and can filter your statements to make sure there are no things like: || 1==1
I would like to start a little talk about this, so if you encountered SQL injection before, would you share it with us, how they did it, or in general how do you prevent SQL injections in your application?
As the winter in the Northern hemisphere is giving way to spring, slowly but surely, blog posts are blooming in the gardens of Oracle, SQL Server and MySQL. This Log Buffer plucks some of them for your reading pleasure.
Providing A Persistent Data Volume to EMC XtremIO Using ClusterHQ Flocker, Docker And Marathon
There is sliced bread in SQL.
Oracle Cloud – Your service is suspended due to exceeding resource quota !
EM12c Compliance ‘Required Data Available’ flag – Understanding and Troubleshooting
How can I see my invisible columns
Auto Generate Your Database Documentation
A Lightweight, Self-adjusting, Baseline-less Data Monitor
Keeping POST and GET Separated
How often should I run DBCC CHECKDB?
Disabling SQL Server Optimizer Rules with QUERYRULEOFF
MySQL Contributions status
Planets9s: Building scalable database infrastructures with MariaDB & HAProxy
High availability with asynchronous replication… and transparent R/W split
mysql_real_connect is not thread safe
Now available in swanhart-tools: NATIVE asynchronous query execution for any MySQL client!