2014년 6월 21일 토요일

Memcached Replication (2)

Modifications

As written in the first article about Memcached Replication, need to change some of MySQL 5.6 source code.
Below diagram will explain how MySQL Memcached plugin works and what I modified.



(Blue line is original behavior of MySQL 5.6 Memcached plugin's Caching and Innodb-only mode, Bold brown line means my modifications)


And the number of the diagram means ...


  1. InnoDB API will transfer the input data to InnoDB engine(Table) as configured with innodb_api_bk_commit_interval, daemon_memcached_r_batch_size, daemon_memcached_w_batch_size system variables. And after that InnoDB API will write changes to binary log. But this is not happened on Cache-only mode. I modified MySQL code as memcached changes is also written to binary log but not apply that changes to InnoDB engine on Cache-only mode Memcached plugin. On original MySQL 5.6 Memcached plugin, dictionary table(innodb_memcache DB) and container table needs for Caching and Innodb-only cache mode not for Cache-only mode. But after modification dictionary table and container table is gotten to need for binary log writting. Of course we will not store memcached data to container table, but need it's template for binary log writting. 
  2. On slave side, SQL thread have to apply relay log gotten by IO thread to MySQL Memcached plugin NOT InnoDB tables. But on original MySQL replication, SQL thread directly access InnoDB API not Memcached. I need a way to access to Memcached plugin. I implemented it as SQL thread bring up another memcached client (using libmemcached client library). I can use Non-blocking mode mode with libmemcached client library for fast relay log applying. Actually in original MySQL 5.6, libmemcached.so shared library also exist on your $MYSQL_HOME/lib/plugin, So I renamed it as "libmemcachedserver.so" so that install libmemcached RPM separately on the same system.
  3. Repication SQL thread act as Memcached client have to run as fast as possible, so modified SQL thread's memcached client connect to Memcached socket file rather than 11211 TCP port. And worked as Non-blocking mode for fast replication. Actually SQL thread will apply relay log event as sync-mode on every 100th request so that slave can check replication apply error.




Another modifications


  • Originally, Memcached plugin never open both of TCP port and Unix domain socket at the same time, But I need both channel.
  • Copy Memcached plugin's status metric to MySQL server's status variables so that we can monitor memcached status with MySQL monitoring tool.InnoDB master thread will copy memcached status metrics to MySQL status variables every second.

mysql> show global status like '%kmc%';
+----------------------------------+--------------+
| Variable_name                    | Value        |
+----------------------------------+--------------+
| Innodb_kmc_connection_structures | 2004         |
| Innodb_kmc_curr_connections      | 2002         |
| Innodb_kmc_curr_items            | 70638969     |
| Innodb_kmc_pointer_size          | 64           |
| Innodb_kmc_threads               | 8            |
| Innodb_kmc_total_connections     | 5090         |
| Innodb_kmc_total_items           | 778120599    |
| Innodb_kmc_bytes                 | 18648687816  |
| Innodb_kmc_bytes_read            | 460650586044 |
| Innodb_kmc_bytes_written         | 497924594841 |
| Innodb_kmc_cmd_get               | 7781206118   |
| Innodb_kmc_cmd_set               | 778120599    |
| Innodb_kmc_evictions             | 498675673    |
| Innodb_kmc_get_hits              | 1043938514   |
| Innodb_kmc_get_misses            | 6737267604   |
| Innodb_kmc_limit_maxbytes        | 21474836480  |
+----------------------------------+--------------+


  • Fix related memory leak of MySQL 5.6.14 and expiration time bug of Memcached plugin.
  • Remove every file sync call for fast processing (if possible).
  • Add expire_max_log_files system variables so that we can control total binary logs' size by file count.(This features only use binary log file's suffix number, so if you use frequent "PURGE LOGS" or other commands which switch binary log file, expire_max_log_files system variables will not work as expected)
  • Add Binlog_purge_failed system status variable for monitoring binary log purge
    • 0 : Okay
    • 1 : Set when MySQL server attempt to remove ACTIVE state binary log.
    • 2 : SET when MySQL server attempt to remove USE state binary log.
  • Add kmc_connect_string system variable so that you can change how SQL thread connect to it's local memcached plugin. You can change it whenever you want because it is dynamic variable. But You have to "STOP/START SLAVE" for applying change.




Usable Memcached Operations

Unfortunately on KMC(Kakao MemCached), Some of memcached operations are not usable. It's because of Memcached replication characteristic (And have not tested it).

Usable operations

  • GET
  • SET
  • ADD
  • DELETE
  • REPLACE


Not implemented or Not tested

  • INCREMENT
  • DECREMENT
  • CAS


Performance

GET operations' performance is same as Original Memcached. But SET/ADD/DELETE operations need to written to disk (Binary log file). And there's a lot of complex processing are involed even though it's not sync mode. So SET/ADD/DELETE performance is dramatically lower than original memcached server.
But general purpose of Memcached server, SET/ADD/DELETE operations is not so many. Once data item is cached on Memcached, then GET operation is performed all the time.
If not (If there's a lot of SET/ADD/DELETE), that memcached server has no effect becuase cache ratio is low. And in this case Memcached server only add overhead (I think).

I also modified SQL thread work as parallel based on row, But MySQL 5.6 multi threaded replication make a lot of sync overhead(CPU overhead) among slave threads and coordinate thread.

MySQL Memcached plugin can process more SET/ADD/DELETE operations than normal MySQL's SQL statements, but you have to consider replication.
SQL thread of slave side is working as single thread, and this might be bottleneck.
On Intel X86 commodity server, I think 10k SET/ADD/DELETE operations are limit. it's the limit of replication (No replication delay status)
So I added expire_max_log_files system variable and Binlog_purge_failed system status variable. If you have enough memory, allocate some of memory for binary log directory using RamFS or RamDisk for SET/DELETE/ADD operation's performance. Added system variable and status variable will help RamFs usage management.





Some limitations


  • Container table name must start with "kmc_".
  • Can't use CAS, INCR and DECR operations.
  • KMC(Kakao MemCached) use two upper bytes of Memcached flags field for originated server_id. So this is not compatible with your memcached client.
  • KMC will convert your DELETE operation to SET operation with 1 second expire (because we can't use flags field on DELETE operation).




Download

https://github.com/kakao/mysql_5.6.14_kmc

댓글 없음:

댓글 쓰기