python opencv 去水印实现

主要利用了 opencv 的 inpaint 方法,实现类似蒙版的效果
http://docs.opencv.org/master/df/d3d/tutorial_py_inpainting.html#gsc.tab=0

效果:注意右下角&左上角
原图

处理后:

'''
string to array
'''
def get_opencv_img_from_string(data):
    arr = np.array(bytearray(data), dtype=np.uint8)
    img = cv2.imdecode(arr, -1)
    return img

'''
array to string
'''
def get_buffer_from_bytes_array(data):
    arr = cv2.imencode(".jpg", data)[1].tostring()
    return arr


def watermark_clean(data, platform):
    img = get_opencv_img_from_string(data)
    height = img.shape[0]
    width = img.shape[1]
    rects = ((width - 169, height - 22, width, height),(1, 1, 164, 21)) #水印区域
    cv2.imshow('a', img)
    mask = np.zeros(img.shape[:2], np.uint8)
    for rect in rects:
        x1, y1, x2, y2 = rect
        cv2.rectangle(mask, (x1, y1), (x2, y2), (255, 255, 255), -1)
        img = cv2.inpaint(img, mask, 10.0, cv2.INPAINT_TELEA) #蒙版
    return get_buffer_from_bytes_array(img)


def show_img(img_arr):
    cv2.imshow('test', img_arr)
    cv.WaitKey(0)

Installing OpenCV 2.4.* with Python Support

Opencv的安装简直让想在图像处理方面试试水的新手抓耳挠腮,今天记录下心酸的安装过程,避免 fall in a same hole.

1.

cd ~/
git clone https://github.com/Itseez/opencv.git
git clone https://github.com/Itseez/opencv_contrib.git

2.

cd ~/opencv
mkdir build
cd build

3.

cmake -D BUILD_SHARED_LIBS=OFF -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..

(这部如果遇到文件 ippicv_linux_20151201.tgz 下载失败的情况,可以手动下载至 opencv/3rdpart/ippiv/downloads/liniux_*/ 下 再作cmake)

4.

make

5.

sudo make install

6.

sudo ln -s /usr/local/lib/python2.7/site-packages/cv2.so {your_python_path}/cv2.so
sudo ln -s /usr/local/lib/python2.7/site-packages/cv.py {your_python_path}/cv.py

Android Studio + OpenCV

The below steps for using Android OpenCV sdk in Android Studio. This is a simplified version of this(1) SO answer.

  1. Download latest OpenCV sdk for Android from OpenCV.org and decompress the zip file.
  2. Import OpenCV to Android Studio, From File -> New -> Import Module, choose sdk/javafolder in the unzipped opencv archive.
  3. Update build.gradle under imported OpenCV module to update 4 fields to match your project build.gradle a) compileSdkVersion b) buildToolsVersion c) minSdkVersion and d) targetSdkVersion.
  4. Add module dependency by Application -> Module Settings, and select the Dependenciestab. Click + icon at bottom, choose Module Dependency and select the imported OpenCV module.
    • For Android Studio v1.2.2, to access to Module Settings : in the project view, right-click the dependent module -> Open Module Settings
  5. Copy libs folder under sdk/native to Android Studio under app/src/main.
  6. In Android Studio, rename the copied libs directory to jniLibs and we are done.

Step (6) is since Android studio expects native libs in app/src/main/jniLibs instead of older libs folder. For those new to Android OpenCV, don’t miss below steps

  • include static{ System.loadLibrary("opencv_java"); } or static{ OpenCVLoader.initDebug(); } (Note: for OpenCV version 3 at this step you should instead load the library opencv_java3.)
  • For step(5), if you ignore any platform libs like x86, make sure your device/emulator is not on that platform.

OpenCV written is in C/C++. Java wrappers are

  1. Android OpenCV SDK – OpenCV.org maintained Android Java wrapper. I suggest this one.
  2. OpenCV Java – OpenCV.org maintained auto generated desktop Java wrapper.
  3. JavaCV – Popular Java wrapper maintained by independent developer(s). Not Android specific. This library might get out of sync with OpenCV newer versions.

Apache Kylin使用

部署或建cube出问题大部分都是环境问题或hadoop hbase 版本问题

首先按照文档中所标注的版本进行部署

  • Hadoop: 2.4 – 2.7
  • Hive: 0.13 – 0.14
  • HBase: 0.98 – 0.99
  • JDK: 1.7+

其中略坑的是 hbase的0.9*版本是不支持Hadoop 2.7,的,若hadoop是2.7.*,需要部署hbase 1.* ,对于hbase 1.* 版本需要下载单独编译的kylin二进制包

Binary Package (for running on HBase 1.1.3 or above)

创建Cube时执行 job 出错

1.
native snappy library not available: SnappyCompressor has not been loaded.

原因是hadoop native lib少了snappy解压缩库

sudo yum install snappy snappy-devel
sudo ln -s /usr/lib64/libsnappy.so $HADOOP_HOME/lib/native/libsnappy.so

在 $HADOOP_HOME/etc/hadoop/hadoop-env.sh 增加

export JAVA_LIBRARY_PATH="/usr/local/hadoop/lib/native"

重启

$HADOOP_HOME/sbin/stop-all.sh
$HADOOP_HOME/sbin/start-all.sh

2.

2016-02-22 16:24:16,740 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
	at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:174)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
	... 8 more

解决方法
http://stackoverflow.com/questions/34449561/hadoop-map-reduce-job-class-org-apache-hive-hcatalog-mapreduce-hcatinputformat

The issue here is Kylin assumes the same Hive jars on all Hadoop nodes. And when certain node missing the Hive jars (or even in different location), you get the ClassNotFoundException on HCatInputFormat.

Btw, you should be able to get a clear error message from Yarn job console. This is a met issue.

Deploying Hive to all cluster nodes can surely fix the problem, like you have tried.

Or another (cleaner) workaround is manually configure Kylin to submit Hive jars as additional job dependencies. See https://issues.apache.org/jira/browse/KYLIN-1021

Finally there's also a open JIRA suggests that Kylin should submit Hive jars by default. See https://issues.apache.org/jira/browse/KYLIN-1082

3.

org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:111)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:111)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
        ... 4 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.yarn.conf.YarnConfiguration.getServiceAddressConfKeys(Lorg/apache/hadoop/conf/Configuration;)Ljava/util/List;
        at org.apache.hadoop.yarn.conf.HAUtil.getConfKeyForRMInstance(HAUtil.java:239)
        at org.apache.hadoop.yarn.conf.HAUtil.getConfValueForRMInstance(HAUtil.java:250)
        at org.apache.hadoop.yarn.conf.HAUtil.getConfValueForRMInstance(HAUtil.java:262)
        at org.apache.kylin.job.common.MapReduceExecutable.getRestStatusCheckUrl(MapReduceExecutable.java:191)
        at org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:135)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
        ... 6 more

拷贝新版本的hadoop-yarn-api.*.jar 至hbase/lib  hbase版本不对

4.

kylin 报错: org.apache.hadoop.hbase.TableNotFoundException: Table KYLIN_* is not currently available.
           Load HFile to HBase Table failed

查看hbase log,看了下是snappy的问题,把kylin的压缩方式改为gzip,重启好了,可以顺利建cube了

5.提示 kylin_metadata 已存在 不能在hbase中创建表,然而hbase中又看不到该表

删除zookeeper 上hbase下该表node

./zkClient.sh

rmr /hbase-unsecure/table/kylin_metadata