Identification of character-type picture verification code

 

Identification of character-type picture verification code


Overview

Demand

Program Design

  • Operating system: CentOS7
  • Programming language: Python3 (flask)
  • Reptile: Selenium
  • Image recognition: Tesseract-ocr,pillow,jtessboxeditor
  • Key technology points: Verification Code identification
  • baselineTESSERACT-OCR (Lang:eng): Accuracy rate 200/376 = 53.19%
  • Improvement Programme(1) Image point noise reduction; (2) sample training based on tesseract training tools
  • Improved resultsTraining sample 30–accuracy rate: 302/392=77.04%; Training sample 90–accuracy: 300/352=85.23%

Python-related configuration

Request the installation of the appropriate package for the service

Flask


pip install Flask

Request Library


pip install requests

Selenium Library Configuration

Installing the Selenium Library


pip install selenium

Install Firefox drivers


https://github.com/mozilla/geckodriver/releases

Unzip the Geckodriver file into the/usr/sbin/directory (note that the Firefox version matches, update the Firefox command: Yum install Firefox)

Image Verification Code Identification package installation

pip install pytesseract
pip install PILLOW

TESSERACT-OCR Compiling and installing

(1) Leptonica compile and install

Download Address:

http://www.leptonica.com/download.html

Compile and install:

tar zxvf leptonica-1.75.3.tar.gz
cd leptonica-1.75.3
./configure 
make&&make install

(2) Tesseract compile and install

wget https://github.com/tesseract-ocr/tesseract/archive/tesseract-3.05.01.tar.gz
tar zxvf tesseract-3.05.01.tar.gz
cd tesseract-3.05.01
./autogen.sh
make
make install
ldconfig

Error problem and solution: reference https://it.baiked.com/system/ops/2291.html

Whether the test was successful:


tesseract

Verification Code Identification

Crawl the corresponding page

# Introduce relevantpackage
from PIL import Image,ImageEnhance
import pytesseract
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import ui
from selenium.webdriver.firefox.options import Options

# Crawler Crawl Code: Slightly

Get CAPTCHA image

def get_picture(driver):
    global cache
    if cache == []:
        captchaElem = driver.find_element_by_xpath('//*[@id="captcha_img"]')
        captchaX = int(captchaElem.location['x'])
        captchaY = int(captchaElem.location['y'])
        captchaWidth = captchaElem.size['width']
        captchaHeight = captchaElem.size['height']
        captchaRight = captchaX + captchaWidth
        captchaBottom = captchaY + captchaHeight
        cache = [captchaX,captchaY,captchaRight,captchaBottom]
    else:
        captchaX,captchaY,captchaRight,captchaBottom = cache

    driver.get_screenshot_as_file("screenshot.png")
    imgObject = Image.open("screenshot.png")

    im = imgObject.crop((captchaX, captchaY, captchaRight, captchaBottom))
    
    return im

Image preprocessing

def im_process(im):
    # WillRGBColor Map to Grayscale
    gray = im.convert('L')
    # Will灰度图按照设定阈值转化为二值图
    gray = gray.point(lambda x: 0 if x < 100 else 1, '1')
    
    return gray

Image de-noising

After converting to a two-value picture, you need to clear the noise.
The image of this project is relatively simple, most of the noise is also the simplest kind of isolated point, so you can detect these outliers can be removed a lot of noise.
about how to remove more complex noise and even interference lines and color blocks, there are more mature algorithms: Flood filling method Flood Fill, this paper for the sake of simplification, the use of a simple nine-Gongge denoising method:

Count the black dots inside the nine gongge around a black dot.;

If the black dot is less than2To prove that this point is an outlier.,And get all the outliers.;

Bulk removal of all outliers at once。

The specific implementation algorithm is as follows:

## Noise reduction
def sum_9_region_new(img, x, y):
    '''Determine the noise point '''
    cur_pixel = img.getpixel((x, y))  # The value of the current pixel point
    width = img.width
    height = img.height
 
    if cur_pixel == 1:  # If the current point is a white area,The neighborhood value is not counted
        return 0
 
    # Because there are black dots all around the current picture.,So the surrounding black spots can be removed
    if y < 2:  # In this example,The first two lines of black dots can be removed
        return 1
    elif y == height - 1:  # The bottom line
        if x < 1 or x == width - 1:
            return 1
        else:
            sum = img.getpixel((x - 1, y - 1)) \
                      + img.getpixel((x - 1, y)) \
                      + img.getpixel((x, y - 1)) \
                      + cur_pixel \
                      + img.getpixel((x + 1, y - 1)) \
                      + img.getpixel((x + 1, y)) 
            return 6 - sum 
    else:  # yNot at the border
        if x < 2:  # First two columns
            return 1
        elif x == width - 1:  # Right Non-vertex
            sum = img.getpixel((x - 1, y - 1)) \
                  + img.getpixel((x - 1, y)) \
                  + img.getpixel((x - 1, y + 1)) \
                  + img.getpixel((x, y - 1)) \
                  + cur_pixel \
                  + img.getpixel((x, y + 1)) 
            return 6 - sum
        else:  # Have9of the field conditions
            sum = img.getpixel((x - 1, y - 1)) \
                  + img.getpixel((x - 1, y)) \
                  + img.getpixel((x - 1, y + 1)) \
                  + img.getpixel((x, y - 1)) \
                  + cur_pixel \
                  + img.getpixel((x, y + 1)) \
                  + img.getpixel((x + 1, y - 1)) \
                  + img.getpixel((x + 1, y)) \
                  + img.getpixel((x + 1, y + 1))
            return 9 - sum

 
def collect_noise_point(img):
    '''Collect all the noise points'''
    noise_point_list = []
    for x in range(img.width):
        for y in range(img.height):
            res_9 = sum_9_region_new(img, x, y)
            if (0 < res_9 < 3) and img.getpixel((x, y)) == 0:  # Find the Orphan Point
                pos = (x, y)
                noise_point_list.append(pos)
    return noise_point_list
 
def remove_noise_pixel(img, noise_point_list):
    '''Location information according to the noise point,Eliminate the black point noise of binary images'''
    for item in noise_point_list:
        img.putpixel((item[0], item[1]), 1)

Verification Code Identification

# Parsing validation Image characters
def parse_ycode(im):
    # Pretreatment
    im = im_process(im)
    # Noise removal
    noise_point_list = collect_noise_point(im)
    remove_noise_pixel(im, noise_point_list)
    #  pytesseractIdentification Verification Code
    yanzhengma = pytesseract.image_to_string(im,lang='eng')
    yanzhengma = list(filter(str.isalnum, str(yanzhengma)))

    if len(yanzhengma) == 5:
        return yanzhengma       
    else:
        im.save(SAVE_FILE+'0-'+''.join(yanzhengma)+'.jpg')
        return None
        

TESSERACT-OCR Training

Install dependent packages


yum install cairo-devel pango-devel libicu-devel

Compiling and installing training tools

Tesseract root directory, execute the following command:

./configure
make training
make training-install

Installing Jtessboxeditor

Jtessboxeditor requires version support for JRE7 (Java Runtime environment).

After installing the JRE, download Jtessboxeditor, unzip, run the Train.bat file to run.

Specific operation can be referred to:

https://www.jianshu.com/p/5c8c6b170f6f

Compositing pictures

Return to the win system and run the Jtessboxeditor tool to synthesize all the pictures in a. tif format (named [Lang].
[Fontname].exp[num].tif].

Generate a Box file

Open a command line in the same directory as the TIF file, generating the corresponding box file (*.box) to generate a box file that records each word and its location coordinates identified by Tesseract.


tesseract [lang]. [fontname].exp[num].tif [lang]. [fontname].exp[num] batch.nochop makebox

Correcting text content

put [Lang]. [Fontname].exp[num].box downloaded, put it under the win system and put it before [Lang].
[Fontname].exp[num].tif directory.
Use Jtessboxeditor to start correcting text.

The specific correction operation can refer to:

https://www.jianshu.com/p/5c8c6b170f6f

Start training

Replace the original box file with the modified box file.
Perform the following actions:

tesseract [lang]. [fontname].exp[num].tif [lang]. [fontname].exp[num] box.train.stderr

This step generates two text files:

[Lang]. [FontName]. exp [num]. tr

And

[Lang]. [FontName]. exp [num]. txt

, the latter has only some newline characters, which correspond to the shape information of each character in the box file in the TIF picture file, the way of recording is actually to treat a character as a polygon, and the TR file records the position, direction, length of each edge of the polygon.

Generating Character Set information


unicharset_extractor [Lang]. [FontName]. exp [num]. Box

This step generates a text file named Unicharset, which, as its name suggests, records a character set that holds the character information that is not duplicated in the box file, Fu Jian one line per individual character.

Create a font information file

We can train tesseract to recognize different fonts in the same language (only one font is trained here) and we need to provide font-related features, which are indicated by a text file called Font_properties.
Each line of this file records the information for a font in the following format:

<fontname><italic> <bold> <fixed> <serif> <fraktur>
</fraktur></serif></fixed></bold></italic></fontname>

A font named boldface is used in the training of this article, so font_properties requires a line of font information that begins with boldface.

In addition to the manual creation of this file, TESSERACT-OCR source also provides a such font_properties file (training/langdata/font_properties), and there is already a lot of font information, Therefore, it is not allowed to manually create the following steps to use the file, the direct designation of the use of this file is OK.
For example:

Boldface 0 1 0 0 0

Polymerization

First Use shapeclustering


shapeclustering -F font_properties -U unicharset [lang]. [fontname].exp[num].tr

This step will output a file named Shapetable, and the next mftraining will automatically load the file in the current directory.

Next Execute mftraining


mftraining -F font_properties -U unicharset -O [lang].unicharset [lang]. [fontname].exp[num].tr

The output has a warning, but does not affect, three files are generated when the execution is completed: [Lang].unicharset, Inttemp, pffmtable

Final execution cntraining


cntraining [Lang]. [FontName]. exp [num]. tr

This step generates a file named Normproto.

Merging build Traineddata files

Now you just need to merge all the files (shapetable, Normproto, Inttemp, pffmtable) and rename them with the same prefix, such as Lang.


combine_tessdata lang.

Note: Don’t forget the last point!

Place the generated [Lang].traineddata in the Tessdata directory.
Then you can test it in the language of your training:

tesseract --list-langs
tesseract image.tif output -l lang 

Hibernate website Download source and support jar

Visit our website:

Hibernate website website

Or get in here?

Write a picture description here

The latest stable version in this

Write a picture description here

click Download

Write a picture description here

A few seconds to start downloading, the slow speed of the patient point

Write a picture description here

Historical version, as if the calendar is not so obvious, is the first few versions

Write a picture description here

Next up is the source.

Write a picture description here

Look here, click, OK

Write a picture description here

That’s a compliment to the previous tutorial.

Build SONAR+JENKINS+MAVEN+SVN complete Continuous integration Code Analysis management




Build SONAR+JENKINS+MAVEN+SVN complete Continuous integration Code Analysis management



 



1. Build sonar Server (5.3 version here)

Preconditions: With JDK environment
Unzip to a disk


Start the sonar server

Enter the Sonarqube-5.3/bin directory according to the current system environment into the appropriate directory to launch the corresponding script file, here take Win64 as an example into the sonarqube-5.3/bin/windows-x86-64 double-click Startsonar.bat (

Linux environment remember sonar.sh and wrapper files simultaneously authorized chmod 777

To


Configure the sonar server to display Chinese

Copy to Directory Sonarqube-5.3/extensions/plugins


Configure Sonar server JDBC

Open the catalog file sonarqube-5.3/conf/sonar.properties
Join the data source configuration (MySQL is used here to support multiple databases):
sonar.jdbc.url=jdbc:mysql://192.168.1.8:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance
sonar.jdbc.username=deploy_user
sonar.jdbc.password=deploy_pass
sonar.sorceEncoding=UTF-8
sonar.login=admin
sonar.password=admin
Note Restart the sonar server to complete the above operations, the sonar server configuration is complete
Access Address:


http://localhost:9000



(The sonar server default port 9000,sonar.properties can be modified)


2. Build sonar Client for parsing code (2.5 version here)

Unzip to a disk


Configuring the Sonar client JDBC and server addresses

Open the catalog file sonar-scanner-2.5/conf/sonar-runner.properties
Join the data source configuration (MySQL is used here to support multiple databases):
sonar.jdbc.url=jdbc:mysql://192.168.1.8:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance
sonar.jdbc.username=deploy_user
sonar.jdbc.password=deploy_pass
Join the Sonar server address (the analysis results can be synchronized to the server):


To configure environment variables:

SONAR_RUNNER_HOME=D:\sonar\sonar-scanner-2.5
Path=%SONAR_RUNNER_HOME%\bin;
Open cmd input sonar-runner-version
Appears as above indicates a successful configuration


3. Perform Code Analysis tasks

Create a new file at the root of a project sonar-project.properties copy the following:
# must be unique in a given SonarQube instance
sonar.projectKey=lkpg
# this is the name displayed in the SonarQube UI
sonar.projectName=anjbo-lkpg
sonar.projectVersion=2.0
# Path is relative to the sonar-project.properties file. Replace “\” by “/” on Windows.
# Since SonarQube 4.2, this property is optional if sonar.modules is set.
# If not set, SonarQube starts looking for source code from the directory containing
# the sonar-project.properties file.
sonar.sources=src
# Encoding of the source code. Default is default system encoding
#sonar.sourceEncoding=UTF-8
As shown in the following illustration:
Enter CMD in the current directory address bar into the cmd command window
Execute the command Sonar-runner to enter the analysis steps:
When you open the sonar server, the analyzed items appear:


4. Build Jenkins (5.3 versions here)


Download complete Double click next to complete

Enter Jenkins for the first time to enter a password, password location:/root/.jenkins/secrets/initialadminpassword

Go to the Select Plugin installation screen and select the first one (install suggested plugins)
 
 

After the plug-in installation is complete, the first user needs to be created, and it is recommended to use test or admin
 

Once you’ve created a user, you’re ready to use Jenkins.
 

Click Start using Jenkins to enter Jenkins


5. System settings


MAVEN project configuration

SonarQube Servers Configuration

Where Server authentication token is generated on sonar server

Award-generated tokens copied to


Server Authentication token can

Note Open the SCM, or it will appear later

ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging. ERROR: SonarQube scanner exited with non-zero code: 1Finished: FAILURE


6、Global Tool Configuration


Use local setting files to build your own central repository
Using a locally built JDK environment
Using a locally built sonar client
Use the local Maven,maven build process:

Unzip to a directory:

Configuring Environment variables
MAVEN_HOME=D:\Java\apache-maven-3.0.5
Path=%MAVEN_HOME%\bin;
Entering cmd input mvn-v appears as follows to indicate a successful configuration:


7, plug-in management

To search for the required plug-in, check the installation directly:


8, new task completion of automatic compilation packaging and Code analysis

Enter the project name and select the MAVEN project
Configure SVN information, credentials first come in need add SVN account password
Configure the release steps to add a sonar parser, and of course you can perform the analysis task before the pre steps compilation
Add post Build action to publish compiler to server container
Click on the right icon to start building
Go to console output log
Build and Code Analysis complete

Windows Install git client

1, the client installation tools are as follows

  • Git-2.12.2.2-64-bit.exe: Is the git real tool that needs to be installed
  • Tortoisegit-2.4.0.2-64bit.msi: It is a visual operation tool for Git, if you do not install Tortoisegit, all GIT operations must use DOS command

2. Installation Instructions

  • The above two tools are installed by default (that is, one-click installation, no need to modify any settings, direct next)

3. Configure User Name

  • Method One: Right mouse button-git Bash here, configured with the following code
git config --global user.name "Tom"    //User name:Take the code cloud as an example,该网站上的User name
git config --global user.email "[email protected]"    //Mailbox:Take the code cloud as an example,该网站上的Mailbox

  • Method Two: Right mouse button-tortoisegit-settings-git (

    Friendly tip: After installing the Tortoisegit visualization tool, you can use method two

  • Check whether the configuration was successful: Enter the code and query results as follows

git config --list

 

JSP page at the beginning of the error resolution

A lot of people in the creation of JSP end will encounter the following error, thieves a few annoying, I was forced to many times have encountered this error, the first time I encountered the time I do not know how to do.

Although for the function can be implemented, but look at the JSP page error is very uncomfortable.

Error code:

<%@ page language=”java” contentType=”text/html; charset=UTF-8″

pageEncoding=”UTF-8″%>

Cause of error: no tomcat files.

I am a MAVEN project and the Web project should be the same.

1. Right-click Maven Dependencies, select Build Path, and then select Configure Build Path.

2. Right-click the project name to select Properties and select Java Build Path.

Select Add Library … Select Server Runtime, click Next, click on your own tomcat configured in Eclipse, then click Finish.
You can do it.

Basically there is no problem with this step, but the index.jsp in my WebApp directory still has an error.
If you’re right on top, you don’t have to look down.

1. Right-click Maven Dependencies, select Build Path, and then select Remove from Build path.
Kill this package lib.

2. Right-click the project name to select Properties, select Maven, and then select Java EE Integration.
The top of the Enable Project Specifis settings hook, will pop up a dialog box, OK, then apply the application, then OK.

Complete these operations, the JSP page will not be an error

“Cocos Creator Basic Tutorial (Others)” – Publishing games with Github Pages

First you have to have a good game.

Click on the item under the menu bar-“Build the release

Publishing Platform Select Web Desktop (for PC-side access)

Write a picture description here

Click Build and then click Run, depending on the effect of the operation

Adjust resolution

, and try to make the entire game appear on a full page.

Like this

Write a picture description here

Publish the platform and change it to Web Mobile (for mobile Access)

Write a picture description here

Note the device orientation and the initial scene

Click Build and then click Run, Chrome browser Windows Press F12,mac Press Command+option+i, or the top right corner-“more tools-” developer tools, open Developer tools, click on the phone Chart preview phone effect

Write a picture description here

Click on the Open button, go to the release path, found that there are two built-in packages, then we put these two packages separately to the computer and mobile side.

Write a picture description here

This assumes you didn’t use GitHub.

Download, install git


https://git-scm.com/

Sign up for GitHub


https://github.com/

If we don’t get these three steps, let’s just give up the line.

Remember account password

Open Git’s own git bash or the system’s own terminal

Input

git version

Show version number to install Git successfully

git config --global user.name "What's your name?" //Set User name 
git config --global user.email "Your [email protected]" //Set up a mailbox

Go to GitHub website, the upper right corner plus a new warehouse, the warehouse name is guaranteed to be this format “your account name. Github.io”, only this warehouse can use the GitHub pages feature.

Write a picture description here

Enter the user name. Github.io the Setup interface of this warehouse, pull down to the following, choose a topic casually.

Write a picture description here

Write a picture description here

Copy the warehouse link (you should have nothing if you are a new warehouse)

Write a picture description here

Terminal.

git clone Warehouse Links

Copy the Web-desktop and Web-mobile packages from the first build to the warehouse directory

and renamed it Star-catcher and Star-catcher-m.

Write a picture description here

Terminal input

git add .
git commit -m "Add a star-picking game"
git push -u origin master

You need to enter a password to lose

Write a picture description here

Finally, you can access the game through the warehouse name plus the game path.

It’s mine


https://potato47.github.io/star-catcher/


https://potato47.github.io/star-catcher-m/


Github Pages also has a lot of features to ponder on its own.

Springmvc+spring+mybatis’s Comprehensive Exercise 2 (build MAVEN environment)

2.1 Setting up local maven in eclipse

The version that Maven chooses is 3.
I installed one in the local and set it up in eclipse.
Local maven in Eclipse


Maven configuration file

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <localRepository>E:\GreenAPP\apache-maven-3.3.9\repo</localRepository>
    <mirrors>
        <mirror>
            <id>alimaven</id>
            <name>aliyun maven</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
            <mirrorOf>central</mirrorOf>
        </mirror>
    </mirrors>
    <profiles>
        <profile>
            <id>jdk-1.8</id>
            <activation>
                <activeByDefault>true</activeByDefault>
                <jdk>1.8</jdk>
            </activation>
            <properties>
                <maven.complier.source>1.8</maven.complier.source>
                <maven.complier.target>1.8</maven.complier.target>
                <maven.complier.complierVersion>1.8</maven.complier.complierVersion>
            </properties>
        </profile>
    </profiles>
</settings>

2.2 Creating a Maven project

2.2.1 Step by step

Directly above the picture.


Step 1


Create a MAVEN project Step 1


Step 2


Write a picture description here

2.2.2 Fix Error

There are two places that are not quite right,

1. Version of the JRE

2. The Web. xml file and the folder for the website project are missing.

Write a picture description here

The repair process is as follows:

1. Select the item, right-click, select Properties, open the dialog box. Modify the Java version.
and remove the dynamic WEB module tick.

Write a picture description here

2. Check the dynamic Web Module again and select version 3.0.
Click further configuration available … to open the dialog box.

Write a picture description here

3. Enter the folder, tick Generated Web. XML deployment descriptor, and click OK.

Write a picture description here

4. You can see the generated Web. xml file and the corresponding folder.

Write a picture description here

2.3 Writing Pom

2.3.1 Add method

In Maven’s

Warehouse

Search for the repository to be added, select the version, drag the repository information directly into the Pom file on the details page.
Each time you save the Pom file, the project automatically refreshes the jar package.
Write a picture description here

2.3.2 The jar package that needs to be added to the POM:

  • spring-webmvc : 4.3.13.RELEASE
  • spring-jdbc: 4.3.13.RELEASE
  • spring-aspects: 4.3.13.RELEASE
  • spring-test: 4.3.13.RELEASE [test]
  • mybatis : 3.4.5
  • mybatis-spring : 1.3.1
  • mybatis-generator-core : 1.3.5
  • mysql-connector-java : 5.1.41
  • c3p0 : 0.9.1
  • jstl : 1.2
  • servlet-api : 3.0
  • junit : 4.12 [test]

These are the current projects needed to build, with the continuation of the project, and then continue to improve and complement the Pom file.

2.4 Pom File

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.hh</groupId>
    <artifactId>ssm</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>war</packaging>
    <dependencies>


        <!-- https://mvnrepository.com/artifact/org.springframework/spring-webmvc -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-webmvc</artifactId>
            <version>4.3.13.RELEASE</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.springframework/spring-jdbc -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-jdbc</artifactId>
            <version>4.3.13.RELEASE</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.springframework/spring-aspects -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-aspects</artifactId>
            <version>4.3.13.RELEASE</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.springframework/spring-test -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>4.3.13.RELEASE</version>
            <scope>test</scope>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.mybatis/mybatis -->
        <dependency>
            <groupId>org.mybatis</groupId>
            <artifactId>mybatis</artifactId>
            <version>3.4.5</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.mybatis/mybatis-spring -->
        <dependency>
            <groupId>org.mybatis</groupId>
            <artifactId>mybatis-spring</artifactId>
            <version>1.3.1</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/c3p0/c3p0 -->
        <dependency>
            <groupId>c3p0</groupId>
            <artifactId>c3p0</artifactId>
            <version>0.9.1</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.41</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/jstl/jstl -->
        <dependency>
            <groupId>jstl</groupId>
            <artifactId>jstl</artifactId>
            <version>1.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/javax.servlet/javax.servlet-api -->
        <!-- Attention:Use SpringMVC The simulation test,Need servlet 3.0 -->
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
            <scope>provided</scope>
        </dependency>


        <!-- https://mvnrepository.com/artifact/junit/junit -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.mybatis.generator/mybatis-generator-core -->
        <dependency>
            <groupId>org.mybatis.generator</groupId>
            <artifactId>mybatis-generator-core</artifactId>
            <version>1.3.5</version>
        </dependency>
    </dependencies>
</project>

So far, the basic environment of the project has come to an end.
The following is the configuration of the framework.

(No hurry, slow down, meticulous, remember firmly)

The simplest Springcloud tutorial in history | Sixth: Distributed Configuration Center (Spring Cloud Config) (Finchley version)

Reprint please indicate the source:


http://blog.csdn.net/forezp/article/details/81041028


This article is derived from

Fang Zhibong’s Blog

In the previous article about Zuul, it has been mentioned that the configuration service is used to store the configuration files for each service.
It’s spring Cloud Config.

First, Introduction

In distributed systems, because of the large number of services, in order to facilitate the unified management of service profiles, real-time updates, so the need for distributed configuration Center components. In spring cloud, there is a distributed Configuration center component, Spring Cloud Config, which enables configuration services to be placed in the memory of the configuration service (that is, local) and also in a remote Git repository.
In Spring Cloud Config component, there are two roles, one is config server and the other is config client.

Second, build config Server

Parent MAVEN project omitted, parent Pom file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.forezp</groupId>
    <artifactId>sc-f-chapter6</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>pom</packaging>

    <modules>
        <module>config-server</module>
        <module>config-client</module>
    </modules>

    <name>sc-f-chapter6</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.3.RELEASE</version>
        <relativePath/>
    </parent>


    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <spring-cloud.version>Finchley.RELEASE</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>


</project>

Create a Spring-boot project, named Config-server, whose pom.xml is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.forezp</groupId>
    <artifactId>config-server</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>config-server</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>com.forezp</groupId>
        <artifactId>sc-f-chapter6</artifactId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>


</project>

At the entrance of the program application class plus the @enableconfigserver annotation opens the function of the configuration server, the code is as follows:


@SpringBootApplication
@EnableConfigServer
public class ConfigServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(ConfigServerApplication.class, args);
    }
}

You need to configure the following in the program configuration file Application.properties file:

spring.application.name=config-server
server.port=8888

spring.cloud.config.server.git.uri=https://github.com/forezp/SpringcloudConfig/
spring.cloud.config.server.git.searchPaths=respo
spring.cloud.config.label=master
spring.cloud.config.server.git.username=
spring.cloud.config.server.git.password=


  • Spring.cloud.config.server.git.uri: Configuring the Git repository address
  • Spring.cloud.config.server.git.searchPaths: Configuring the Warehouse Path
  • Spring.cloud.config.label: Configuring a branch of a warehouse
  • Spring.cloud.config.server.git.username: User name to access the GIT repository
  • Spring.cloud.config.server.git.password: Access the user password for the GIT repository

If the Git warehouse is a public warehouse, you can not fill in the user name and password, if the private warehouse needs to fill out, this example is open warehouse, rest assured that use.

Remote Warehouse

https://github.com/forezp/SpringcloudConfig/

There is a file in the Config-client-dev.properties file that has a property:

foo = foo version 3

Startup program: Access

http://localhost:8888/foo/dev

{"name":"foo","profiles":["dev"],"label":"master",
"version":"792ffc77c03f4b138d28e89b576900ac5e01a44b","state":null,"propertySources":[]}

proves that the Configuration service center can obtain configuration information from remote programs.

The HTTP request address and resource file mappings are as follows:

  • /{application}/{profile}[/{label}]
  • /{application}-{profile}.yml
  • /{label}/{application}-{profile}.yml
  • /{application}-{profile}.properties
  • /{label}/{application}-{profile}.properties

Third, build a config client

Re-create a springboot project named Config-client, whose pom file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.forezp</groupId>
    <artifactId>config-client</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>config-client</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>com.forezp</groupId>
        <artifactId>sc-f-chapter6</artifactId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-config</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>


</project>

Its configuration file

bootstrap.properties

spring.application.name=config-client
spring.cloud.config.label=master
spring.cloud.config.profile=dev
spring.cloud.config.uri= http://localhost:8888/
server.port=8881
  • Spring.cloud.config.label indicates the branch of the remote repository
  • spring.cloud.config.profile

    • Dev Development environment configuration file
    • Test environment
    • Pro Formal environment
  • spring.cloud.config.uri=

    http://localhost:8888/

    Specifies the URL of the Configuration service center.

The entry class of the program, write an API interface “/hi”, return the value of the Foo variable read from the configuration center, the code is as follows:

@SpringBootApplication
@RestController
public class ConfigClientApplication {

    public static void main(String[] args) {
        SpringApplication.run(ConfigClientApplication.class, args);
    }

    @Value("${foo}")
    String foo;
    @RequestMapping(value = "/hi")
    public String hi(){
        return foo;
    }
}

Open URL Access:

http://localhost:8881/hi

, the page displays:

foo version 3

This means that config-client gets the properties of Foo from Config-server, and Config-server is read from the Git repository, as shown in figure:

Azure (2).png

This article source code download:


https://github.com/forezp/SpringCloudLearning/tree/master/sc-f-chapter6

Iv. references


http://blog.csdn.net/forezp/article/details/70037291


http://cloud.spring.io/spring-cloud-static/Finchley.RELEASE/single/spring-cloud.html

Springboot+vue front and back end separation, perfect handling of permissions problems with springsecurity (i)

In the current back-end separation, the handling of permissions issues is also a little different from our traditional approach. I was in charge of a project Rights Management module A few days ago, now the Rights Management module has been done, I would like to pass 5-6 articles, to introduce the problems encountered in the project and my solution, I hope this series can give small partners some help. This series of articles is not a hands-on tutorial, mainly introduces the core ideas and explains the core code, complete code small partners can be on GitHub on the star and clone down to study.
In addition, originally planned to run the project to the Internet for small partners to view, but before the server in order to save money, memory only 512M, two applications do not run up (there is already a
V Tribe Open Source project

Running), so the small partners can only take a look at the screenshot below, at the end of the article has a deployment tutorial, deployed to the local can also see the full effect.

In this paper, we will not talk about the implementation, I would first introduce my Authority management module, a whole structure, and the final rendering of the effect.

Database design

The rights database mainly contains five tables, namely Resource table, role table, user table, Resource Role table, User Role table, database relation model as follows:

Write a picture description here

On this table, I say the following points:

The 1.hr table is the user table that holds the user’s basic information.

2.role is the role table, the Name field represents the English names of the roles, according to the Springsecurity specification, will be

ROLE_

Start, the Namezh field represents the Chinese name of the role.

The 3.menu table is a resource table, which involves a bit more fields, because my front end uses Vue, so when the user logs in successfully, the system will dynamically load the required modules according to the user’s role, and the information of all modules will be saved in the menu table, the path in the menu table, Fields such as component, Iconcls, KeepAlive, Requireauth, and so on are all fields required in vue-router, meaning that the data in the menu will be returned to the front end in JSON, and then dynamically updated by Vue router,
The menu also has a field URL that represents a URL pattern, which is a path matching rule, assuming that a path matching rule is
/admin/**

, then when the user initiates a

/admin/user

The request, will be

/admin/**

Intercept, the system then to see what the role of this rule, and then to see if the user has the appropriate role, and then determine whether the request is legitimate.

The following figure is part of the data in the user table, the role table, and the Resource table (database scripts can be downloaded in the project address at the end of the article, where

resources/vhr.sql

):

Write a picture description here

Write a picture description here

Write a picture description here

Overall effect

First of all, different users will see different system menus according to different roles after successful login, the complete menu is as follows:

Write a picture description here

When different users sign in, they may see differences, as follows:

Write a picture description here

Each user’s role is assigned by the system administrator, and the page where the system administrator assigns roles to users is as follows:

Write a picture description here

The system administrator can also manage resources that can be manipulated by different roles, as follows:

Write a picture description here

Other deletions, searches and other trivial features I will not be introduced here.

Project Address

As a result of the commercial agreement, the original project can not be shared to all the small partners, so I specifically made an open source project, the overall function of the project is more, but considering this series of articles is mainly to introduce the Rights Management module, so other modules are I temporarily castrated, but the small partners can rest assured that The Code of the Rights Management module is not deleted, and the code and data related to rights management are complete and can be run directly.
The small partner will log in to the backend system as an administrator, then click on
System Management-> Basic information Set-> Permission Group

To configure the resources that can be manipulated by different roles, and then tap

System Management-> Operator management

, you can manage the role of each operator.

Project Address:

https://github.com/lenve/vhr

Rapid deployment

1.clone Project to Local

[email protected]:lenve/vhr.git

2. Database scripts are placed in the resources directory of the Hrserver project and database scripts are executed in MySQL

3. Database configuration in the Application.properties file under the resources directory of the Hrserver project

4. Run the Hrserver project in IntelliJ idea


OK, at this point, the server started successfully, at this time we directly in the address bar input

http://localhost:8082/index.html

Can access our projects, if you want to do two development, please continue to see the fifth to sixth step.

5. Go to the Vuehr directory and enter the following command in the command line:

# Installation dependencies
npm install

# In localhost:8080 Start Project
npm run dev

Since I have configured port forwarding in the Vuehr project, forwarding the data to Springboot, so after the project is started, enter

http://localhost:8080

We have access to our front-end projects, and all requests pass the data to Springboot via port forwarding (note that you do not close the Springboot project at this time).

6. Finally, you can open the Vuehr project with Webstorm and other tools, continue to develop, after the development is completed, when the project to go online, still enter the Vuehr directory, and then execute the following command:

npm run build

After the command executes successfully, a dist folder is generated under the Vuehr directory, and the two files in that folder are copied to the resources/static/directory in the Springboot project by copying the static and index.html of the folders.
Then it can be accessed directly like the 4th step.


Step 5 requires you to Nodejs, NPM and other have a certain use of experience, not familiar with the small partners can search the first self-study, recommended

Vue Official Tutorials


Precautions

Again, this is just a Rights management function module, only the Rights management function is complete after running.
Small partners after successful local deployment, you can modify the role of each user and the resources that each role can manipulate, after the successful modification, log off and log in as the modified user, you can see the menu changes.

OK, this article we will first say so much, from the next article, I will share with the small partners this effect to achieve the core ideas, welcome the small partners continue to pay attention.

Follow the public number, you can receive the latest articles in time:

Write a picture description here

Installation of Visdom and visualization application of loss function under Pytorch

2018/9/18 Update feel TENSORBOARDX plug-in better use, has been diverted

https://github.com/lanpa/tensorboardX

Update: The new visdom0.1.7 installation method is:

conda install -c srivasv visdom

Pytorch can be used as a visualization tool under the Visidom

1. Installation

pip install visdom
conda install visdom

Start

python -m visdom.server

In the browser input: http://localhost:8097/, you can start


Note:

(1) Sometimes after installation can not be used, such as the Web page nothing, you can try to replace anaconda3\lib\site-packages\visdom under the static folder (if you use anaconda words), because of the network, cause the file download is not complete,
File Link Link:
https://pan.baidu.com/s/1c4liqoK

Password: Q1SX, under Linux (Ubuntu) is also the same processing, under Lib\site-packages\visdom Replace the static file, restart the effective!!!!

(2) Sometimes when you start Visdom, the display target computer refused to wait for the network connection is unsuccessful, the following test can be used

from visdom import Visdom
viz = Visdom()
assert viz.check_connection()

Mainly I was using the agent to surf the internet, which led to the change of Internet options for my Internet Explorer, as shown below:

2. Example of a visual loss function

 

First introduce the line () function in Visdom:

1) Draw a straight line

from visdom import Visdom
import numpy as np
viz = Visdom(env='my_wind')#The name of the Set environment window is'my_wind',If you do not set the namemainIn
tr_loss=list(range(100))
viz.line(Y=np.array(tr_loss), opts=dict(showlegend=True))

Before running the above program, first start Visdom, enter Python-m visdom.server in cmd, and then enter in the browser: http://localhost:8097/, the default is in the main environment:

The above code, we set the name of the environment window is ‘ my_wind ‘, so we need to open the My_wind environment, as follows:

Run the Code!

2) If you want to draw more than one line

 
from visdom import Visdom
import numpy as np
viz = Visdom(env='my_wind')#The name of the Set environment window is'my_wind',If you do not set the namemainIn
tr_loss=list(range(100))
ts_loss=list(range(10,110))
viz.line(Y=np.column_stack((np.array(tr_loss),np.array(ts_loss))), opts=dict(showlegend=True))

Note: The values of the above variables are determined before they are plotted.

3) If you want to draw the values that gradually occur with the program, as in training, you can use the line Update method

from visdom import Visdom
import numpy as np
viz = Visdom(env='my_wind')
x,y=0,0
win = viz.line(
    X=np.array([x]),
    Y=np.array([y]),
    opts=dict(title='two_lines'))
for i in range(10):
    x+=i
    y+=i
    viz.line(
        X=np.array([x]),
        Y=np.array([y]),
        win=win,#winTo remain consistent
        update='append')


Finally, the loss function is visualized in the process of deep learning training, and the reference is

Pytorch Practical Guide

Visual operations in the

Visualize the loss function in the visual_loss.py file:

#coding:utf8
import visdom
import time
import numpy as np

class Visualizer(object):
    def __init__(self, env='default', **kwargs):
        self.vis = visdom.Visdom(env=env, **kwargs)
        self.index = {}         
    def plot_many_stack(self, d):
        '''
        self.plot('loss',1.00)
        '''
        name=list(d.keys())
        name_total=" ".join(name)
        x = self.index.get(name_total, 0)
        val=list(d.values())
        if len(val)==1:
            y=np.array(val)
        else:
            y=np.array(val).reshape(-1,len(val))
        #print(x)
        self.vis.line(Y=y,X=np.ones(y.shape)*x,
                    win=str(name_total),#unicode
                    opts=dict(legend=name,
                        title=name_total),
                    update=None if x == 0 else 'append'
                    )
        self.index[name_total] = x + 1            

function function test in Jupyter NOTEBOOK–LOSS_VISUAL_TEST.IPYNB:

from visual_loss import Visualizer
from torchnet import meter
#Use torchnetTo store the loss function,If not,Please installconda install torchnet
'''
Pre-training models、Loss Function Settings 
vis = Visualizer(env='my_wind')#To visualize the added content
loss_meter = meter.AverageValueMeter()#To visualize the added content

for epoch in range(10):
    #EachepochBefore you begin,will be stored in thelossClear,Start recording again
    loss_meter.reset()#To visualize the added content
    model.train()
    for ii,(data,label)in enumerate(trainloader):     
        ...
        out=model(input)
        loss=...
        loss_meter.add(loss.data[0])#To visualize the added content
        
    #lossVisualization of
    #loss_meter.value()[0]Returns the storedlossThe mean value
    vis.plot_many_stack({'train_loss': loss_meter.value()[0]})#To visualize the added content    
'''
#Example
vis = Visualizer(env='my_wind')#To visualize the added content
loss_meter = meter.AverageValueMeter()#To visualize the added content
for epoch in range(10):
    loss_meter.reset()#To visualize the added content
    loss_meter.add(epoch)#Assumeloss=epoch
    vis.plot_many_stack({'train_loss': loss_meter.value()[0]})#To visualize the added content 
    #If you want to show it at the same timetest loss,Followed suit,并Use字典的形式赋值,As follows。You can also display bothtrainAndtest accuracy
    #vis.plot_many_stack({'train_loss': loss_meter.value()[0],'test_loss':test_loss_meter.value()[0]})#To visualize the added content 

File structure:

where __init__.py is empty


Note: My visdom version is

I do not know which one to use, through the Conda install Visdom installed version is older, the latest version of the official website is

The installation method is in the terminal input

conda install -c srivasv visdom

For more details, see

Website

© 2018 Yang's site

Theme by Anders NorénUp ↑