Ranger2.4+OpenLdap 用户权限管理

介绍

使用openldap做统一用户管理
使用ranger做用户权限管理
对Hdfs,Hive,Spark,Trino,Yarn进行权限控制

openLdap用户

执行以下命令，添加用户，用户组

组	用户	用户	用户
shuguo	liubei	guanyu	zhangfei
weiguo	caocao	caopi	caozhi
wuguo	shunquan	shunce	lvmeng

user='zhangfei'  #替换用户名
EXAMPLE_GROUP='shuguo'  #替换用户所属组USER_UID=$((2000+$RANDOM%999))OPENLDAP_ROOT_DN='cn=admin,dc=example,dc=com'
OPENLDAP_ROOT_PASSWORD='Admin1234!'
OPENLDAP_USERS_BASE_DN='ou=users,dc=example,dc=com'
COMMON_DEFAULT_PASSWORD='Admin1234!'
OPENLDAP_BASE_DN='dc=example,dc=com'ldapsearch -D "$OPENLDAP_ROOT_DN" -w $OPENLDAP_ROOT_PASSWORD -b "cn=$EXAMPLE_GROUP,ou=groups,$OPENLDAP_BASE_DN" >& /dev/null
if [ "$?" != "0" ]; thenGROUP_GID=$((3000+$RANDOM%999))echo "用户组不存在 --${GROUP_GID}"
elseGROUP_GID=`ldapsearch -D "$OPENLDAP_ROOT_DN" -w $OPENLDAP_ROOT_PASSWORD -b "ou=groups,$OPENLDAP_BASE_DN" -s sub "(&(objectClass=posixGroup)(cn=$EXAMPLE_GROUP))" gidNumber | grep "^gidNumber:" | awk '{print $2}'`echo "用户组存在 --${GROUP_GID}"
fi# add user
cat << EOF | ldapadd -D "$OPENLDAP_ROOT_DN" -w $OPENLDAP_ROOT_PASSWORD
dn: uid=$user,$OPENLDAP_USERS_BASE_DN
objectClass: posixAccount
objectClass: top
objectClass: inetOrgPerson
uid: $user
displayName: $user
sn: $user
homeDirectory: /home/$user
cn: $user
uidNumber: $USER_UID
gidNumber: $GROUP_GID
userPassword: $(slappasswd -s $COMMON_DEFAULT_PASSWORD)
EOF#检查是否有groups，如果不存在就创建，如果存在就直接把刚刚的用户添加到groups中
ldapsearch -D "$OPENLDAP_ROOT_DN" -w $OPENLDAP_ROOT_PASSWORD -b "cn=$EXAMPLE_GROUP,ou=groups,$OPENLDAP_BASE_DN" >& /dev/null
if [ "$?" != "0" ]; thenecho '不存在，需要创建'cat << EOF | ldapadd -D "$OPENLDAP_ROOT_DN" -w $OPENLDAP_ROOT_PASSWORD
dn: cn=$EXAMPLE_GROUP,ou=groups,$OPENLDAP_BASE_DN
cn: $EXAMPLE_GROUP
objectclass: top
objectclass: posixGroup
gidNumber: $GROUP_GID
memberUid: $user
EOF
elseecho '已经存在添加用户'cat << EOF | ldapmodify -D "$OPENLDAP_ROOT_DN" -w $OPENLDAP_ROOT_PASSWORD
dn: cn=$EXAMPLE_GROUP,ou=groups,$OPENLDAP_BASE_DN
changetype: modify
add: memberUid
memberUid: $user
EOF
fi

查看openldap用户，用户组
ldapsearch -x -D “cn=admin,dc=example,dc=com” -w ‘Admin1234!’ -b “ou=groups,dc=example,dc=com”

# shuguo, groups, example.com
dn: cn=shuguo,ou=groups,dc=example,dc=com
cn: shuguo
objectClass: top
objectClass: posixGroup
gidNumber: 3563
memberUid: liubei
memberUid: guanyu
memberUid: zhangfei# weiguo, groups, example.com
dn: cn=weiguo,ou=groups,dc=example,dc=com
cn: weiguo
objectClass: top
objectClass: posixGroup
gidNumber: 3358
memberUid: caocao
memberUid: caopi
memberUid: caozhi# wuguo, groups, example.com
dn: cn=wuguo,ou=groups,dc=example,dc=com
cn: wuguo
objectClass: top
objectClass: posixGroup
gidNumber: 3847
memberUid: shunquan
memberUid: shunce
memberUid: lvmeng

master sssd节点用户，用户组同步

sudo systemctl restart sssd
sudo sss_cache -ESSD同步成功：
[hadoop@ip-10-0-29-52 ~]$ id liubei
uid=2117(liubei) gid=3563(shuguo) groups=3563(shuguo)
[hadoop@ip-10-0-29-52 ~]$ id guanyu
uid=2664(guanyu) gid=3563(shuguo) groups=3563(shuguo)
[hadoop@ip-10-0-29-52 ~]$ id zhangfei
uid=2937(zhangfei) gid=3563(shuguo) groups=3563(shuguo)[hadoop@ip-10-0-29-52 ~]$ id caocao
uid=2838(caocao) gid=3358(weiguo) groups=3358(weiguo)
[hadoop@ip-10-0-29-52 ~]$ id caopi
uid=2210(caopi) gid=3358(weiguo) groups=3358(weiguo)
[hadoop@ip-10-0-29-52 ~]$ id caozhi
uid=2137(caozhi) gid=3358(weiguo) groups=3358(weiguo)[hadoop@ip-10-0-29-52 ~]$ id shunquan
uid=2689(shunquan) gid=3847(wuguo) groups=3847(wuguo)
[hadoop@ip-10-0-29-52 ~]$ id shunce
uid=2092(shunce) gid=3847(wuguo) groups=3847(wuguo)
[hadoop@ip-10-0-29-52 ~]$ id lvmeng
uid=2215(lvmeng) gid=3847(wuguo) groups=3847(wuguo)

重启ranger-sync服务

sudo ranger-usersync restart

ranger-admin查看同步成功

hue手动添加用户组

测试场景

测试用户：

只给liubei用户权限

测试组：

给weiguo组权限

测试role：

创建role，并绑定guanyu用户,绑定wuguo组，然后给role权限

Hadoop 插件

hadoop fs -mkdir /ranger-test

hadoop fs -chown hadoop:hadoop /ranger-test

hadoop fs -chmod 700 /ranger-test

该文件夹除了hadoop用户以外都没有权限查看

未授权：

用户

用户liubei 权限被禁止

组

weiguo组中用户caocao 权限被禁止

Role

用户guanyu

wuguo组中shunquan 权限被禁止

授权：

sudo ./disable-hdfs-plugin.sh

sudo ./enable-hdfs-plugin.sh

插件可能需要重装一下，重启服务

用户

用户liubei 权限被允许

组

weiguo组中用户caocao 权限被允许

Role

用户guanyu权限被允许

wuguo组中shunquan 权限被允许

Hive 插件

未授权：

将ranger权限全部清空

用户

用户liubei 权限被禁止

组

weiguo组中用户caocao 权限被禁止

Role

用户guanyu权限被禁止

wuguo组中shunquan 权限被禁止

授权：

sudo ./disable-hive-plugin.sh
sudo ./enable-hive-plugin.sh
重启一下hive相关服务

用户

用户liubei 权限被允许

组

weiguo组中用户caocao 权限被允许

Role

用户guanyu

wuguo组中shunquan 权限被允许

Spark 插件

未授权：

将ranger权限全部清空

用户

用户liubei 权限被禁止

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user liubei

组

weiguo组中用户caocao 权限被禁止

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user caocao

Role

用户guanyu权限被禁止

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user guanyu

wuguo组中shunquan 权限被禁止

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user shunquan

授权：

用户

用户liubei 权限被允许

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user liubei

组

weiguo组中用户caocao 权限被允许

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user caocao

Role

用户guanyu权限被允许

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user guanyu

wuguo组中shunquan 权限被允许

spark-sql --master yarn --deploy-mode client --conf 'spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension' --proxy-user shunquan

Trino 插件

未授权：

将ranger权限全部清空

用户

用户liubei 权限被禁止

组

weiguo组中用户caocao 权限被禁止

Role

用户guanyu权限被禁止

wuguo组中shunquan 权限被禁止

授权：

https://blog.csdn.net/qq_36096641/article/details/127518912

注意Trino的权限需要对每一访问层级进行设置，比如

catalog 级别

catalog + schema 级别

catalog + schema + table 级别

并且需要配置用户可访问information_schema这个trino元数据库，这样才能访问其它的表
————————————————

catalog级别：

information_schema元数据库：

catalog + schema 级别：

catalog + schema + table 级别

用户

用户liubei 权限被允许

组

weiguo组中用户caocao 权限被禁止

trino-plugin支持组失败，修改配置/usr/lib/trino/etc/access-control.properties，
请参考：https://blog.csdn.net/qq_40616823/article/details/123008319#:~:text=%E9%80%9A%E8%BF%87debug%E5%8F%91%E7%8E%B0this.useUgi%E9%BB%98%E8%AE%A4%E6%98%AFfalse

添加：ranger.use_ugi=true

Role

用户guanyu权限被允许

wuguo组中shunquan 权限被禁止

trino-plugin支持组失败，修改配置/usr/lib/trino/etc/access-control.properties，
请参考：https://blog.csdn.net/qq_40616823/article/details/123008319#:~:text=%E9%80%9A%E8%BF%87debug%E5%8F%91%E7%8E%B0this.useUgi%E9%BB%98%E8%AE%A4%E6%98%AFfalse

添加：ranger.use_ugi=true

Yarn 插件

yarn配置队列

[{"classification": "yarn-site","properties": {"yarn.scheduler.capacity.queue-mappings-override.enable": "true","yarn.acl.enable": "true"}},{"classification": "capacity-scheduler","properties": {"yarn.scheduler.capacity.queue-mappings": "u:hadoop:default,u:caocao:weiguo,u:liubei:shuguo,u:shunquan:wuguo,u:guanyu:shuguo,u:zhangfei:shuguo,u:shunce:wuguo,u:lvmeng:wuguo,u:caopi:weiguo,u:caozhi:weiguo","yarn.scheduler.capacity.root.queues": "default,shuguo,weiguo,wuguo","yarn.scheduler.capacity.root.capacity": "100","yarn.scheduler.capacity.root.acl_administer_queue": "","yarn.scheduler.capacity.root.acl_submit_applications": "","yarn.scheduler.capacity.root.default.capacity": "10","yarn.scheduler.capacity.root.default.maximum-capacity": "60","yarn.scheduler.capacity.root.default.acl_administer_queue": "hadoop","yarn.scheduler.capacity.root.default.acl_submit_applications": "caocao,liubei,shunquan,guanyu,zhangfei,shunce,lvmeng,caopi,caozhi","yarn.scheduler.capacity.root.shuguo.capacity": "30","yarn.scheduler.capacity.root.shuguo.maximum-capacity": "70","yarn.scheduler.capacity.root.shuguo.acl_administer_queue":"hadoop","yarn.scheduler.capacity.root.shuguo.acl_submit_applications": "","yarn.scheduler.capacity.root.weiguo.capacity": "30","yarn.scheduler.capacity.root.weiguo.maximum-capacity": "70","yarn.scheduler.capacity.root.weiguo.acl_administer_queue": "hadoop","yarn.scheduler.capacity.root.weiguo.acl_submit_applications": "","yarn.scheduler.capacity.root.wuguo.capacity": "30","yarn.scheduler.capacity.root.wuguo.maximum-capacity": "70","yarn.scheduler.capacity.root.wuguo.acl_administer_queue": "hadoop","yarn.scheduler.capacity.root.wuguo.acl_submit_applications": "","yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator"}}
]

配置如下(注意：配置的ranger.add-yarn-authorization=false会无视下面这个配置)：

liubei,guanyu,zhangfei 默认提交到shuguo

caocao,caopi,caozhi 默认提交到weiguo

shunquan,shunce,lvmeng 默认提交到wuguo

但是都没有权限提交，需要ranger授权才行

未授权：

将ranger权限全部清空

并配置

By default, fallback to YARN ACLs are enabled. If access cannot be determined by Ranger policies, authorization will fall back to YARN ACLs. If this behavior needs to be changed, modify YARN plugin config - ranger.add-yarn-authorization.

默认情况下，允许回退到YARN acl。如果访问不能由Ranger策略确定，则授权将返回到YARN acl。如果需要修改此行为，请修改YARN plugin config - ranger.add-yarn-authorization。

禁用yarn默认的ACL，都从ranger-yarn通过

sudo vim /usr/lib/hadoop-yarn/etc/hadoop/ranger-yarn-security.xml

<property><name>ranger.add-yarn-authorization</name><value>false</value>
</property>

sudo systemctl restart hadoop-yarn-resourcemanager.service

提前准备用户在hdfs上的文件

hadoop fs -mkdir /user/liubei
hadoop fs -chown liubei:liubei /user/liubeihadoop fs -mkdir /user/caocao
hadoop fs -chown caocao:caocao /user/caocaohadoop fs -mkdir /user/guanyu
hadoop fs -chown guanyu:guanyu /user/guanyuhadoop fs -mkdir /user/shunquan
hadoop fs -chown shunquan:shunquan /user/shunquan

用户

用户liubei 提交到shuguo权限被禁止

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user liubei \
--queue shuguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

组

weiguo组中用户caocao 提交到weiguo权限被禁止

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user caocao \
--queue weiguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

Role

用户guanyu提交到wuguo 权限被禁止

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user guanyu \
--queue wuguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

wuguo组中shunquan提交到wuguo 权限被禁止

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user shunquan \
--queue wuguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

授权：

授权用户

授权组

授权role

用户

用户liubei 提交到shuguo权限被允许

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user liubei \
--queue shuguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

组

weiguo组中用户caocao 提交到weiguo权限被允许

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user caocao \
--queue weiguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

Role

用户guanyu提交到wuguo 权限被允许

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user guanyu \
--queue wuguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

wuguo组中shunquan提交到wuguo 权限被允许

spark-submit \
--master yarn \
--deploy-mode cluster \
--proxy-user shunquan \
--queue wuguo \
--class  org.apache.spark.examples.SparkPi  /usr/lib/spark/examples/jars/spark-examples.jar

Spark Sql+结合Livy

由于spark sql在hue上通过livy进行提交

因此：配置sudo vim /etc/livy/conf/livy.conf

livy.impersonation.enabled: true

使得根据对应的hue用户初始化session环境

tail -f /var/log/livy/livy-livy-server.out

由于初始环境的时候，并没有指定配置

–conf ‘spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension’ ，因此并不能管住权限

修改sudo vim /etc/spark/conf/spark-defaults.conf，添加配置

spark.sql.extensions org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension

重新初始化session，并查看spark webui，可以看到配置如下：

未授权

将ranger权限全部清空

用户

用户liubei 提交到shuguo，但是查询被禁止

组

weiguo组中用户caocao提交到weiuo，但是查询被禁止

Role

用户guanyu在livy初始化环境的时候失败，因为guanyu默认提交的队列为shuguo，但是ranger并没有给其权限

wuguo组中shunquan提交到wuguo ，但是查询被禁止

授权

用户

用户liubei 提交到shuguo，并且查询被允许

组

weiguo组中用户caocao提交到weiuo，并且查询被允许

Role

用户guanyu在livy初始化环境的时候失败，因为guanyu默认提交的队列为shuguo，但是ranger并没有给其权限

wuguo组中shunquan提交到wuguo ，并且查询被允许