? i wanna export a table to json format files,but after gging,nothing solutions found.i known,pig is used to do soome sql like mapreduces stuff; and hive is a dataware to build on hbase.but i cant some soutions /wordaround to do that too( maybe i miss something)
? so i consider to use mr to figure out this case.of course,the data in hfile is byte format,that means it must be first converted to string using utf8 then put it into json object to escape something special chars.but wait a moment ,when i go on like this,but i not happy:the output chars limit of 64k chars occurs:
static int writeUTF(String str, DataOutput out) throws IOException {
int strlen = str.length();
int utflen = 0;
int c, count = 0;
/* use charAt instead of copying String to char array */
for (int i = 0; i < strlen; i++) {
c = str.charAt(i);
if ((c >= 0x0001) && (c <= 0x007F)) {
utflen++;
} else if (c > 0x07FF) {
utflen += 3;
} else {
utflen += 2;
}
}
if (utflen > 65535)
throw new UTFDataFormatException(
"encoded string too long: " + utflen + " bytes");
...
? ?maybe u want to use these method to address it ,but no luck also:
write(byte[]) :write bytes directly to fs
writeBytes(sting):write a byte per char to fs
writeChars(string):write two byts per char
?
? that means only the style of writeUTF() is suitable to write out misc type text to file.and it‘s using the utf8 encoding to write bytes.so i think i can construct a ‘json-sytle‘ format bytes to achive this.
? and the utf8 decoding will be the reverse step of this encoding(copy from jdk)
for (;i < strlen; i++){
c = str.charAt(i);
if ((c >= 0x0001) && (c <= 0x007F)) {
bytearr[count++] = (byte) c;
} else if (c > 0x07FF) {
bytearr[count++] = (byte) (0xE0 | ((c >> 12) & 0x0F));
bytearr[count++] = (byte) (0x80 | ((c >> 6) & 0x3F));
bytearr[count++] = (byte) (0x80 | ((c >> 0) & 0x3F));
} else {
bytearr[count++] = (byte) (0xC0 | ((c >> 6) & 0x1F));
bytearr[count++] = (byte) (0x80 | ((c >> 0) & 0x3F));
}
}
?
? ?and we know,the byte[] from hfile is utf8 encodig ,so the workaround below code is present:
private void write2Hdfs() throws IOException {
//write per line to fs
logger.info(" start to write file..");
long start=System.currentTimeMillis();
for(MiniArchive jso : container){ //buffer size has been set in init,5m
this.outputStream.write(‘{‘);
this.outputStream.write(this.BYTE_KEY);
this.outputStream.write(jso.getKey());
this.outputStream.write(this.BYTE_QUOT);
if(jso.getTitle() != null){
this.outputStream.write(this.BYTE_TITLE);
this.outputStream.write(jso.getTitle());
this.outputStream.write(this.BYTE_QUOT);
}
...
? ?yeh,the escape code is must before write out the appropriate bytes
private static byte[] escapeRef(byte[] bytes) {
if(bytes == null || bytes.length == 0)
return null;
Set<Integer> set = null;
// long t1 = System.currentTimeMillis();
for(int i=0;i<bytes.length;i++){
if(bytes[i] == ‘\"‘){
if(set == null){
set = new HashSet<Integer>(10);
}
set.add(i);
}
}
// System.out.println("t1:" +(System.currentTimeMillis() - t1));
if(set != null){
// long ts = System.currentTimeMillis();
byte[] ret = new byte[bytes.length + set.size()];
/*
* 2
*/
// System.out.println("alg cost1:" + (System.currentTimeMillis() - ts));
// ts = System.currentTimeMillis();
int newIndex = 0;
// long countts = 0;
for(int i=0;i<bytes.length;i++){//-does it more effect use arraycopy() for first index?
// long ts2 = System.currentTimeMillis();
// boolean cnt = set.contains(i); //if set is large size,it‘s worse
// countts += System.currentTimeMillis() - ts2;
if(set.contains(i)){
ret[newIndex] = ‘\\‘; //insert
newIndex++;
set.remove((Integer)i); //note:i in list is object
}
ret[newIndex++] = bytes[i];
}
...
?
? ? so i did construct a fake ‘json‘ myself;)
? ?after ?some perf tests,i found that using this solution is about 4x fast than 3 waves conversations:
pseudo code: a.convert bytes retrieved from hbase to string b.put string to json,and get toString() to convert the special char ‘"‘ c.convert this string back to bytes for writing out using write(byte[])
?
hbase-export table to json file
原文：http://leibnitz.iteye.com/blog/2266649