| 
  • If you are citizen of an European Union member nation, you may not use this service unless you are at least 16 years old.

  • Stop wasting time looking for files and revisions. Connect your Gmail, DriveDropbox, and Slack accounts and in less than 2 minutes, Dokkio will automatically organize all your file attachments. Learn more and claim your free account.

View
 

37_hidden_features_of_the_misc_method

Page history last edited by Makoto Inoue 10 years, 8 months ago

miscメソッドの隠し機能

 

Hidden features of the "misc" method

 

ID: 37

creation date: 2009/11/06 23:34

modification date: 2009/11/06 23:34

owner: mikio

Original:

 

Tokyo Cabinetの抽象データベースAPIとTokyo TyrantのリモートデータベースAPIで使えるmiscメソッドには隠し機能がたくさんある。今回はそれを明らかにしてしまおう。

 

"misc" method at TC and TT has full of hidden features. I will unveil some of these hidden gems today.

 

そもそもmiscって何?

 

TCの抽象データベースAPIは、ハッシュ表DBとB+木DBと固定長配列DBとテーブルDBの全てを同一のインターフェイスで扱う機構である(それを継承して独自の挙動を行わせるスケルトンDB機構もある)。また、TTのサーバはTCの抽象データベースを扱うので、TTのリモートデータベースAPIも、ここまでに挙げた全ての種類のデータベースを同一のインターフェイスで利用することができる。

 

So, what is "misc" method by the way?

 

TC has abstract database API which provides common interface among hdb, bd, fdb, and tdb. Since TT uses this API, TT API inherits same interfaces.

 

DBMもしくはkey-valueストレージとして、キーと値のペアをレコードとして格納したり、キーに対応するレコードを取得たり削除したりするだけならばそれでよいのだが、わざわざ別種の具象DBを用意しているのだからそれ以外のこともやりたくなるのが人情だ。特にテーブルDBはkey-valueの原則から大幅に外れるので、独自の操作を許容しないとまともに使うことができない。そこで、具象DB毎に異なる操作を同じAPI(関数シグネチャ)にて無理やり収めるべく、文字列でサブ関数名を指定するとともに、パラメータをリストとして渡して、戻り値をリストとして受け取るという汎用メソッドを実装した。それがmiscメソッドである。

 

This common API is good enough to use simple Key-Value related operations (e.g.: get/put). However, this will be not enough to handle everything each different databases can do. Table DB, especially, is far from key-value principles, so it won't be useful if we do not offer additional methods. To do so, you can use "misc" method to specify sub function name as first argument (in string), and its argument as second argument as list. This will return response as list, too

 

サブ関数の全貌

misc関数のサブ関数に関する公式の仕様書の言及は、「All databases support "put", "out", "get", "putlist", "outlist", "getlist", and "getpart".」というのみで、詳細はソースを見ないとわからない状態である。メールなどで寄せられた要求に基づいてライブラリ側の実装として提供すべきだと判断した機能はmisc関数のサブ関数としてどんどん追加しているので、仕様の記述が追いついていないのがその理由。さらに、個々のユーザの要求に基づく「ad hoc」な機能を全ユーザが把握すべき仕様にしてしまうと学習の負荷が上がってしまうので、敢えて隠しているというのも理由のひとつである。

 

The full list of sub functions.

The API specification documentation simply says "All databases support "put", "out", "get", "putlist", "outlist", "getlist", and "getpart" and all the detail can be only found by actually reading the source code. This is partly because I keep adding new sub functions as requested via email, so I can not keep up updating the specification. Another reason is to avoid the documentation being bloated by exposing every small details.

 

(NOTE from Translator: Which line of which file does it contain the full list? Does anyone have any pointer to help me guide, as I know very little about c, Makoto)

 

しかし、有用な機能が多いのに知れ渡っていないのは勿体ないので、このブログを読んでいるようなTCマニアのあなたにはコッソリ教えてしまおう。以下に名前と機能概要を列挙する。戻り値なしというのは、空リストが返却されることを意味す る 。エラーはNULLで示される。

 

However, it is unfortunate that some of useful functionalities are hidden. Since you must be hard core TC maniac enough to read this blog, I will tell you now. Here are the list of names and their descriptions. "response: none" means it returns empty list. Error will be represented as NULL.

 

• put

◦ Function: Store a record. Overwrites existing record

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg:  key, value

◦ Response: none

 

• putkeep

◦ Function: Store a record. Do nothing if the record already exists

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg:  key, value

◦ Response: none

 

• putcat

◦ Function: Store a record. Append if the  record already exists

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg:  key, value

◦ Response: none

 

• putdup

◦ Function: Store a record. Place duplicating record at back.

◦ DB:BDB

◦ Arg:  key, value

◦ Response: none

 

• putdupback

◦ Function: Store a record. Place duplicating record at front.

◦ DB:BDB

◦ Arg:  key, value

◦ Response: none

 

• out

◦ Function: Delete a record

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg:  key, value

◦ Response: none

 

• get

◦ Function: Get a record

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: Key

◦ Response: Record value

 

• putlist

◦ Function: Add multiple records

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: list of key value

◦ Response: none

 

• outlist

◦ Function: Delete multiple records

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: List of keys

◦ Response: none

 

• getlist

◦ Function: Get multiple record

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: List of keys

◦ Response: List of key & value pairs

 

• getpart

◦ Function: Get partial of a record

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: Key, Starting offset (opt), Length(opt)

◦ Response: The partial of the mating record

 

• iterinit

◦ Function: Initialise internal iterator

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: Key of the starting record(opt)

◦ Response: none

 

• iternext

◦ Function: Get next record from internal iterator

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: none

◦ Response: Key & Value of the record

 

• sync

◦ Function: Sync database to file

◦ DB:HDB, BDB, FDB, TDB

◦ Arg: none

◦ Response:none

 

• optimize

◦ Function: Optimize database

◦ DB:HDB, BDB, FDB, TDB

◦ Arg: Tuning Parameter string 

◦ Response: none

 

• vanish

◦ Function: Delete all record

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: none

◦ Response: none

 

• error

◦ Function: Get an last occurred error

◦ DB:HDB, BDB, FDB, TDB

◦ Arg: none

◦ Response: String consisting of error number and error message

 

• defrag

◦ Function: Defrag database

◦ DB:HDB, BDB, FDB, TDB

◦ Arg: number of steps (opt)

◦ Arg: none

 

• cacheclear

◦ Function: Delete all database cache

◦ DB:HDB, BDB, TDB

◦ Arg: none

◦ Response: none

 

• regex

◦ Function: Get multiple records which matches regular expression

◦ DB:MDB, NDB, HDB, BDB, FDB, TDB

◦ Arg: Regular Expression, Max number of records to return (opt)

◦ Response:  List of Key & Value of the record

 

• range

◦ Function: Get multiple records within range. Lower range will not be included.

◦ DB:NDB, BDB, FDB

◦ Arg: Upper range, Max number, Lower range

◦ Response:  List of Key & Value of the record

 

• setindex

◦ Function: Set an index

◦ DB:TDB

◦ Arg: column name, type

◦ Response: none

 

• search

◦ Function: Search table

◦ DB:TDB

◦ Arg:List of "addcond\0<name>\0<op>\0<expr>" , "setorder\0<name>\0<type>" ,"setlimit\0<max>\0<skip>" 

◦ Response: List of sring which matches the condition

 

• metasearch

◦ Function: Metasearch (or)

◦ DB:TDB

◦ Arg: List of "search" conditions, delimited by "next"

◦ Response: List of sring which matches the condition

 

• genuid

◦ Function: Generate unique ID

◦ DB:TDB

◦ Arg: non

◦ Response: Generated ID

 

多機能なのもmiscの特徴だが、もうひとつの特徴として、TTから呼ぶ際にレプリケーションをするか否かを選べることがある。デフォルトでは操作履歴が更新ログに書き込まれてレプリケーションされるが、RDBMONOULOGオプションをつけて呼び出すと更新ログの記録もレプリケーションも行われない。なので、検索や値の取得など、データベースの更新を伴わない操作ではRDBMONOULOGをつけるべきである。標準メソッドのtcrdbsyncやtcrdboptimizeやtcrdbvanishもレプリケーションされるので、そうしたくないが最適化や初期化を行いたい場合にはmiscのサブ関数版をRDBMONOULOG付きで呼ぶとよい。あと、misc関数はTTのLua拡張からも呼べるので、組み合わせ技でかなり面白いことができると思う。

 

One thing worth noting is  that you can choose replication option when you call from TT. By default, all your operation will be written to update logging files. If you call with RDBMONOULOG option, then it won't record update log nor replication. If you are doing get or search operation which does not require replication, you should call with RDBMONOULOG.

 

If you want to initialise or optimise, but not replicate tcrdbsync, tcrdboptimize, tcrdbvanish, then you should call it with RDBMONOULOG, too.

You can do quite a lot if you call "misc" from Lua extensions.

 

まとめ

miscを使いこなすとTCの抽象データベースとTTのリモートデータベースの細かい制御ができて便利なことがある。今後もユーザからの要求の中で合理性のありそうなものは、多少インターフェイスが汚くとも、misc関数の中に取り込んでいくだろう。とはいえ、俺も暇じゃないんで、何でも言うこと聞くわけじゃないよ。

 

Summary

Once you become familiar with "misc" method, you can do a lot more with TC abstraction database and TT. If I get more quests from user to include more sub functions, I may implement (may not have enough time to respond to all request though). 

 

 

Comments (1)

Flinn said

at 7:25 pm on Mar 29, 2010

Re: NOTE from Translator:

The tcadbmisc method starts on line 1556 of tcadb.c in Tokyo Cabinet 1.4.43. The method is a very lengthy switch with lots of conditionals. It appears the above translation documents all commands at this time.

You don't have permission to comment on this page.